acceptodds
Withdrawn from ICLR 2027

CoVisco: Codec-Native Vision Encoder with Native Token Compression for Efficient Unified Visual Understanding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language models face a fundamental scaling bottleneck: the number of visual tokens grows with frames times patches per frame, and long-video understanding quickly becomes computationally prohibitive. Existing remedies compress visual tokens after the vision encoder and before the language model—an inherently lossy post-hoc pruning of a representation that was never trained to be compressed, with particularly severe information loss on video. Recent work has moved compression into encoder pretraining, but does so by discarding most fine-grained tokens outright, conflating compression with deletion. We present \model, a unified image-video codec-native vision encoder that performs native token compression: during pretraining, each video segment is equipped with a small set of learnable abstract tokens that learn to absorb the segment's semantics, while all fine-grained patch tokens retain their full information flow—compression and fine-grained preservation coexist rather than compete. Cross-segment interaction flows exclusively through abstract tokens, providing a long-video modeling scheme that avoids dense cross-frame full attention. For MLLM deployment, a lightweight token selector further enables a family of token strategies that combine abstract tokens with top- fine-grained tokens selected on demand, dynamically trading accuracy for the visual-token budget. In codec mode, the selector operates on the tokens already retained by codec-based input selection, yielding a two-stage compression pipeline before the LLM: codec-based selection first reduces the visual input, while the learned abstract-token interface and selector further reduce the visual sequence exposed to the LLM. For uniformly sampled and frame-collage inputs, the same learned second stage operates directly on the encoder's fine-grained token stream. Together, these components provide multi-level visual modeling: patch tokens preserve fine-grained evidence, abstract tokens encode segment-level semantics, and abstract-mediated interaction builds video-level context. Pretrained with three contrastive objectives on 565M image–text pairs, and 6.4M videos, is competitive with selected larger embedding models on video-oriented evaluation while using a compact vision tower. It retains useful image-side retrieval and classification performance, and compresses 64-frame videos into as few as 400 visual tokens (2.4% retention) while achieving video understanding performance comparable to, and in some cases better than, OneVision-Encoder under the evaluated settings. These results highlight its ability to support efficient unified visual understanding across both images and videos.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision Foundation Model, Unified Vision Understanding, Vision Language Model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_GT2DOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CoVisco, a unified image-video vision encoder designed for efficient visual token compression. It introduces learnable abstract tokens to capture segment-level semantics while preserving fine-grained patch tokens within the encoder. An alternating attention mechanism enables cross-segment communication through abstract tokens, and a lightweight token selector supports flexible compression ratios for downstream MLLMs. Experiments demonstrate promising video understanding performance with substantially fewer visual tokens exposed to the language model.

Critical strengths
  1. The paper addresses an important problem in efficient video understanding. The proposed design of maintaining both abstract and fine-grained tokens is reasonable, providing a flexible trade-off between compact semantic representations and detailed visual information.

  2. CoVisco achieves competitive or better performance than several reference encoders while exposing significantly fewer visual tokens to the LLM. The evaluation also covers image classification, multimodal retrieval, and video understanding, demonstrating the potential of the unified encoder.

Critical weaknesses
  1. The abstract suggests that "existing approaches primarily perform visual token compression after the vision encoder and before the language model". However, substantial prior research has already explored compression before or within vision encoders, as well as within LLMs. Recent approaches such as PVC, EarlyTom, and MeToM further demonstrate that visual token compression has been extensively investigated at different stages of the processing pipeline. Although the paper acknowledges some encoder-internal approaches, its characterization of prior work remains overly broad and does not clearly establish what fundamental limitation CoVisco addresses beyond existing methods.

  2. Insufficient comparisons with recent token compression methods. The experimental comparisons mainly focus on conventional vision encoders such as CLIP and SigLIP, along with OneVision-Encoder. While these baselines are useful for evaluating representation quality, they are insufficient to demonstrate the advantages of the proposed compression architecture. Several more closely related approaches, including PVC (CVPR 2025), EarlyTom (CVPR 2026), MeToM (CVPR 2026), and Video-XL-Pro, have explored efficient visual token compression using different mechanisms. Comparisons with representative recent methods under comparable token and computational budgets would provide stronger evidence for the effectiveness of CoVisco.

  3. The paper introduces several components, including learnable abstract tokens, alternating segmented attention, and a token selector, but their individual contributions are not adequately isolated. In particular, it remains unclear how much improvement comes from abstract-mediated cross-segment communication, whether the proposed token selector outperforms simpler selection strategies, and how the number of abstract tokens affects performance.

  4. Although codec-native processing is emphasized in the paper's title and motivation, the proposed abstract-token architecture and segmented attention mechanism do not fundamentally depend on codec information. The same architecture supports uniform frame sampling, which achieves comparable or even better performance on several benchmarks. Consequently, the specific contribution of codec-based processing remains unclear, particularly given its close relationship to OneVision-Encoder. A more systematic comparison between codec-based and uniform sampling under matched computational and token budgets would help clarify its role and benefits.

[1] PVC: Progressive Visual Token Compression for Unified Image and Video Processing in Large Vision-Language Models

[2] EarlyTom : Early Token Compression Completes Fast Video Understanding

[3] MeToM: Metadata-Guided Token Merging for Efficient Video LLMs

[4] Video-XL-Pro: Reconstructive Token Compression for Extremely Long Video Understanding

Questions for authors

I encourage the authors to clarify the distinctions between CoVisco and recent encoder-internal compression methods, include stronger and more relevant baselines, and conduct targeted ablations to isolate the contributions of the proposed components. In particular, demonstrating the independent benefits of codec-native processing and the advantages of abstract-token compression under controlled computational budgets would substantially strengthen the paper.

Reviewer_BbzkOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes CoVisco, a unified image-video vision encoder that performs native token compression during pretraining. Unlike prior methods that rely on post-hoc token pruning or discard fine-grained tokens, CoVisco introduces learnable abstract tokens to capture segment-level semantics while preserving fine-grained patch information. Cross-segment interactions are handled exclusively through abstract tokens, enabling efficient long-video modeling without dense cross-frame attention. Extensive experimental results demonstrate its effectiveness in supporting efficient, unified visual understanding across images and videos.

Critical strengths

The paper is well constructed and easy to follow.
The paper provides comprehensive experimental evaluations across several widely used benchmarks.

Critical weaknesses
  1. Unclear distinction from CLS tokens. To what extent do the proposed abstract tokens serve a similar role to the CLS token in CLIP-style models? It would be valuable to evaluate whether applying the proposed attention reduction and downstream MLLM integration strategies directly to CLS tokens could achieve comparable results.
  2. Limited gains despite improved efficiency. As a pretraining-focused work emphasizing efficiency, CoVisco benefits from supporting higher visual resolutions and longer text sequences. However, it still fails to outperform CLIP on ImageNet-1K, raising questions about whether the efficiency gains translate into better representation quality.
  3. Limited scalability evaluation. The paper would benefit from experiments with larger-scale ViT backbones to demonstrate the scalability and effectiveness of the proposed approach.
  4. Weak image understanding performance and incomplete comparisons. Although CoVisco achieves promising results on video understanding tasks, its performance on image understanding tasks remains noticeably behind mainstream visual encoders. Moreover, results under directly comparable configurations, such as CoVisco (Codec, 504, 0.4) and CoVisco (Codec, 504, 1.0), are missing, making it difficult to assess its effectiveness under fair comparison settings.
Questions for authors

Please see weakness

Reviewer_xL68OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents CoVisco, a unified image and video vision encoder that introduces learnable abstract tokens for each temporal segment. Abstract tokens receive contrastive supervision and provide a compact representation, while patch tokens remain available for optional downstream selection. Alternating intra-segment and abstract-mediated attention is intended to reduce dense cross-segment patch interactions. A learned selector exposes abstract-only or abstract-plus-top-K tokens to a fixed language model. The paper evaluates retrieval, image and video benchmarks, and multimodal understanding with a common Qwen3-4B-Instruct-2507 language backbone.

The main contributions of this paper are as follows: 1. A compact representation learned inside the ViT, with the patch stream retained. 2. Alternating layers that route cross-segment information only through abstract tokens. 3. A selector on the frozen encoder that gives the language model abstract-only, abstract + top-K or full-token input.

Critical strengths
  1. The segmented attention scheme gives a plausible global communication path through abstract tokens while avoiding direct all-to-all attention between patches in different segments. The paper distinguishes encoder-side attention cost from the LLM-side token budget.

  2. Abstract-only inference gives promising results. At 400 tokens it matches or beats OneVision-Encoder (10,368 tokens) on 4 of 7 video benchmarks, one of them by only 0.1, and beats SigLIP2 on all 7.

  3. K can change at deployment without retraining, abstract-only inference stays stable across 4, 8 and 16 segments, and encoder-side and LLM-side budgets are kept separate.

  4. The paper also reports transfer beyond video understanding, including image retrieval/classification and visual-document tasks, and states several limitations rather than presenting the method as uniformly superior.

Critical weaknesses
  1. The main claim, that native compression improves on post-hoc compression, is never tested directly. No results table includes a post-hoc method at a matching token budget.

  2. Table 3 is not controlled. Appendix D describes “matched sampling and token budgets”, but resolutions and retention differ, frame counts are not listed, and no uncertainty is reported. In addition, it has no full-token video baseline.

  3. There is no architectural ablation and no measured efficiency, even though “efficient” is in the title.

  4. The fine-grained path and selector have weak support. Patch outputs have no direct loss, and the selector is trained after the ViT is frozen, close to the mismatch Section 1 criticizes. The selector has no random-selection baseline and does not see the question.

Questions for authors
  1. In Table 3, what are the frame and token counts for each column, and how were baselines trained and tuned? Section 3.1 gives T_seg = 32 but frame-based evaluation uses 16, so which applies?

  2. Can you provide ablations of the main components? How does abstract-only inference compare with a post-hoc method at 400 tokens and with full-token CoVisco on video?

  3. What are the end-to-end latency and peak memory at representative video lengths, compared with a dense-attention baseline?

  4. What loss and K budgets train the selector, and how does it compare with random selection? What evidence shows the patch stream keeps fine-grained information, and how do selected tokens convey position?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.