acceptodds
Under review as a conference paper at ICLR 2027

BOTTLENECKED SEMANTIC COMPRESSION BY REPRESENTATION RECONSTRUCTION

Abstract

Vision Transformers incur high computational costs when processing long visual token sequences. Existing reduction methods typically rank tokens independently and discard those deemed unimportant or redundant. We propose a Reconstruction-guided framework for visual token condensation(ReConV), which that summarizes the complete visual sequence with a compact Summary Bank (SMB) and retains tokens whose information is not adequately captured by the SMB, as indicated by their reconstruction residuals. At each compression layer, a lightweight position-conditioned decoder reconstructs the current tokens, leaving shared information compressed in the SMB while explicitly preserving poorly reconstructed details. Experiments on image classification and multimodal understanding demonstrate favorable performance–efficiency trade-offs: ReConV surpasses the uncompressed baseline with \(48%\) fewer GMACs on ImageNet and enables reusable image-level representations for efficient multi-turn inference. Code is available at TokenDrop.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Visual Token Reduction, Token Reconstruction, Image Classification, Multimodal Understanding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_cqj6OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ReConV, a reconstruction-guided framework for visual token compression. A learnable Summary Bank aggregates visual features through cross-attention, and a position-conditioned decoder reconstructs individual token features from their assigned summary slots. Orthogonal reconstruction residuals determine which original or residual tokens are retained alongside the summary tokens. The model is trained with task and contrastive reconstruction objectives. Experiments cover ImageNet classification and LLaVA-based multimodal understanding, including multi-turn evaluation, token-budget ablations, and latency measurements.

Critical strengths
  1. Coherent combination of summaries and explicit details. The framework combines a compact summary representation with tokens carrying information that is poorly explained by that representation. Its prompt-independent compression allows the same image representation to be reused across questions, making the design relevant to repeated-query and multi-turn applications.

  2. Evidence of practical feasibility across tasks. The experiments include both image classification and multimodal understanding, together with component and token-budget ablations. The reported results suggest that the framework can substantially reduce computation while preserving a useful fraction of the original model’s capability.

Critical weaknesses
  1. The novelty relative to closely related reconstruction-guided compression is insufficiently established. ApET: Approximation-Error Guided Token Compression for Efficient VLMs (CVPR 2026) already reconstructs visual tokens from a small basis and uses approximation errors to retain poorly explained tokens. Its scores are conditioned on the selected basis, which also weakens the manuscript’s broad distinction between existing independent scoring and its own summary-conditioned selection. ReConV introduces architectural differences, including a learned Summary Bank, a position-conditioned decoder, and residual representations. However, the paper does not discuss this close prior work or establish the additional capability or benefit of these changes over the existing reconstruction-guided principle. The current component ablations provide partial support but do not isolate this incremental contribution relative to a linear reconstruction baseline.

  2. The experimental comparisons do not sufficiently isolate the benefits of the proposed compression design. Table 2 uses 79.8% as the uncompressed ImageNet baseline, whereas Table 5 reports 80.61% for the authors’ locally trained vanilla model. Appendix B.1 also specifies 300 training epochs for vanilla models and 500 for compressed models, while the main compressed models use distillation. These differences complicate the claim of matching or exceeding uncompressed accuracy with 48% fewer GMACs. In the multimodal experiments, comparable training and adaptation conditions across methods are not clearly established. Table 7 additionally reports dev-set results for ReConV on MMB/MMB-CN while giving different uncompressed baseline values in its caption and first row and incorporating literature results. Relative normalization does not by itself establish comparability across these settings. These issues limit attribution of the reported gains to the compression mechanism.

Questions for authors
  1. What is the substantive contribution beyond reconstruction-error-guided compression as exemplified by ApET? Please explain what the learned summary bank, decoder, and residual representation add, and provide a controlled comparison or existing evidence that isolates this benefit at a fixed total token budget.

  2. Which uncompressed ImageNet baseline corresponds to the models in Table 2: 79.8% or the locally trained 80.61% model? How do the different training durations and distillation settings affect the claim of matching or exceeding the baseline with 48% fewer GMACs?

  3. For Tables 3 and 7, which methods use comparable model checkpoints, training data, and adaptation budgets? Please clarify the evaluation splits and corresponding uncompressed baselines for MMB/MMB-CN, and explain whether the conclusions hold when these conditions are aligned.

  4. The comparisons provide limited coverage of recent state-of-the-art visual token compression methods, particularly learned compression approaches.

Reviewer_TbcaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ReConV, a reconstruction-guided framework for visual token compression. The method uses a learnable Summary Bank (SMB) to aggregate visual information and a position-conditioned decoder to reconstruct token representations. Reconstruction residuals then guide the retention of tokens whose information is not adequately captured by the summary. The framework is evaluated on ImageNet classification and multimodal understanding with LLaVA-1.5-13B, including multi-turn inference in which compressed image representations can be reused across questions.

Critical strengths

Well-organized presentation. The paper is well structured, with a coherent progression from motivation to methodology and empirical evaluation. The architectural illustrations and algorithmic description help communicate the roles of summary aggregation, reconstruction, and residual-guided token selection.

Applicability to both classification and multimodal understanding. The experiments demonstrate that the framework can support both image classification and multimodal understanding while reducing computational requirements. These results suggest that the combination of compact summaries and complementary retained tokens has utility beyond a single task setting.

Clear description of the experimental setup. The paper describes the main datasets, model configurations, training objectives, and evaluation settings, with additional implementation details and ablations in the appendix. These descriptions provide a useful foundation for reproducibility, although several training and initialization details require further clarification.

Critical weaknesses
  1. The multimodal comparisons do not adequately control for training and adaptation. ReConV uses the 558K alignment dataset and 665K instruction dataset to train its compression pipeline and adapt the LLM through LoRA. However, Table 7 does not explicitly include the fine-tuned VisionZip variant, which adapts the multimodal projector using one-tenth of the LLaVA-1.5 data. Its reported results exceed those of ReConV on several overlapping benchmarks, although evaluation protocols must be aligned before drawing definitive conclusions. Moreover, even the VisionZip results included in Table 7 have higher reported average relative performance than ReConV at both 128 and 64 tokens. These observations raise questions about whether the proposed mechanism provides sufficient benefit to justify its additional training requirements. Comparisons under matched data, adaptation budgets, and evaluation protocols are needed to separate the contribution of the compression mechanism from that of task-specific training.

  2. Cross-model transferability and adaptation costs remain insufficiently evaluated. All multimodal experiments are conducted on LLaVA-1.5-13B. Evaluation across multiple benchmarks establishes task coverage within this model, but does not establish applicability across different multimodal architectures.

  3. Both classification and multimodal evaluations omit recent, directly relevant baselines. Table 2 compares only against methods from 2023 or earlier, without evaluating subsequent approaches such as LookupViT and ImagePiece. Table 7 includes several 2025 methods but omits representative alternatives such as PruneSID (ICLR 2026), VisPruner (ICCV 2025), CDPruner (NeurIPS 2025), and DivPrune (CVPR 2025). These methods are relevant not merely because they are recent, but because they address semantic coverage, redundancy, or subset diversity, which are central to the paper’s motivation. Furthermore, the 32-token setting in Table 7 contains only ReConV, leaving its competitiveness under aggressive compression unestablished.

Questions for authors

Does the proposed method offer any advantages over the fine-tuned version of VisionZip in terms of benchmark performance, training cost, or inference efficiency?

Reviewer_nTfrOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses the quadratic cost of self-attention in vision transformers (ViTs) and multimodal LLMs (MLLMs). This cost is manageable for small images, but it quickly becomes a major bottleneck for high-resolution images, multiple images, or videos.

Previous methods reduce the number of visual tokens by scoring, merging, or clustering them. In MLLMs, most methods rank visual tokens using attention scores from the vision encoder or the LLM, and then drop / merge the low-ranked ones. The authors argue this is suboptimal for several reasons: attention shows where the model looked but not what would be lost if a token were removed, reading attention scores can interfere with fast attention kernels, pruning inside the LLM cannot save compute in the LLM layers that come before it, and finally, both independent scoring and merging can lose fine details: the first keeps redundant tokens over subtle ones, the second blurs them.

To this end, the paper proposes ReConV, which compresses the visual tokens before the LLM and without any prompt conditioning. At each compression layer, it works in three steps:

  1. summarize: a small, learned summary bank (8-32 tokens) gathers the content of all visual tokens through cross-attention
  2. rebuild: each visual token is assigned to one summary token, and a small decoder tries to rebuild it from that summary token and the token's position
  3. compare: tokens that are rebuilt well are already captured by the summary and are dropped, while a fixed fraction of the most poorly rebuilt tokens contain missing details and are kept.

For the MLLM, the final output of ReConV is the summary tokens plus the kept tokens, and the compressor is trained with the task loss plus a contrastive reconstruction loss.

The method is tested on image classification (DeiT-S on ImageNet) and on an MLLM (LLaVA-1.5-13B). On ImageNet, with distillation, it reaches 79.9% accuracy with 48% less compute, matching the published DeiT-S result (79.8%). On multi-turn visual question answering with 95% of visual tokens removed, it gives the best accuracy among compressed methods, but only slightly beats random pruning (70.9% vs. 69.8%, uncompressed 80.0%). Classification ablations (Table 6) also show that the reconstruction objective matters much more (removing the decoder costs about 4 points) than the residual-based selection, which is only 0.7 points better than random dropping.

Critical strengths

The idea of keeping only the tokens that a small learned summary fails to reconstruct is a sensible alternative to scoring tokens independently, which directly targets the redundancy problem of top-k selection.

The design is also practical, the propoes ReConV runs before the LLM and uses neither the prompt nor encoder or LLM attention scores, so it is compatible with fast attention kernels and easy to plug into existing inference pipelines.

Critical weaknesses
  • Writing and presentation: the method is simple to describe, but understanding the paper and the way the authors describes it takes several readings and overcomplicates it. The paper also contains many careless errors: placeholder links, inconsistent citation styles and duplicate references, broken figure references, figures that disagree with the equations, and arithmetic mistakes in the result tables. For a method paper, this makes the contribution hard to assess and to reproduce.

  • Limited novelt: the individual components of the proposed method are not new. For example, the summary bank is a set of learned queries that cross-attend to the visual tokens, as in the Q-Former [3] and Perceiver [4]. Keeping the tokens that a small set of basis tokens approximates poorly was already proposed in ApET [5]. The paper's contribution therefore reduces to combining existing ideas, and it also does not show that this combination adds much beyond its parts.

  • Unfair comparisons and missing baselines: the method trains its compressor, the projector, and LoRA weights, while the prior MLLM methods it is compared against are largely training-free. For a fair comparison, I think the authors should also add methods that train their compressors with the same token count and training recipe, such as pixel shuffle, average pooling, a Q-Former, or fine-tuned VisionZip [6]. Since none of these baselines is reported, I think it is hard to contribute the gains to the method itself rather than simply to training a compressor.

  • The results do not support the main claims: on ImageNet, ReConV is trained longer and with distillation, but it is compared against the published DeiT-S rather than the authors' own, stronger baseline. Against that baseline, the claim of surpassing the uncompressed model does not hold. In the ablations, the authors also show that many of the main componenets of the method do not matter: residual-based selection barely beats random dropping, removing positional information from the decoder has no effect, and models that keep only the summary bank perform about as well as those that also keep residual tokens. As for MLLMs, ReConV trails most training-free baselines on single-question and it barely beats random pruning in the multi-turn setting.

  • Evaluation setup: the paper is motivated by induced cost of visual tokens when processing high-resolution images, multiple images, or video, but the authors only evaluate their method on small images: 224px ImageNet & single-image LLaVA-1.5-13B (which is an outdated model by now). There are no larger ViTs, no modern MLLMs, and no high-resolution or video settings.

References

  • [1] How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites, arXiv 2024.
  • [2] Building and Better Understanding Vision-Language Models: Insights and Future Directions, arXiv 2024.
  • [3] BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models, ICML 2023.
  • [4] Flamingo: a Visual Language Model for Few-Shot Learning, NeurIPS 2022.
  • [5] ApET: Approximation-Error Guided Token Compression for Efficient VLMs, CVPR 2026.
  • [6] VisionZip: Longer is Better but Not Necessary in Vision Language Models, CVPR 2025.
Questions for authors

Clarifications

  • How does an uncompressed DeiT-S trained with the same recipe as ReConV (same epochs, teacher, and EMA) perform?
  • What does keeping residual tokens add over keeping only the summary bank, at the same compute? How much do results vary across seeds?
  • Were the Table 3 baselines (random pruning, spatial pooling) trained like ReConV? How does ReConV compare to simple trained compressors, such as pixel shuffle or average pooling, at the same token budget?
  • Does the residual-based selection tend to favor high-norm tokens, and does normalizing the residual by token norm change the results?

Suggestions

  • As mentioned in the weaknesses, I think more evaluations on high-resolution images, multiple images, and video, which motivate the paper, are necessary to demonstrate the effectiveness of the method. The same applies to larger ViTs and more recent MLLMs.
  • Similarly, comparisons against strong trained baselines are needed to show that the gains come from the method itself rather than from training a compressor. Examples include pixel shuffle, average pooling, a Q-Former, or fine-tuned VisionZip, trained with the same token budget and recipe.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.