acceptodds
Under review as a conference paper at ICLR 2027

ViCoS: Verify-then-Correct Retrieval and Context Mosaic for Aggressive Video Token Compression

Abstract

Long-form video question answering requires vision-language models (VLMs) to process thousands of visual tokens, making inference expensive due to the quadratic cost of self-attention. Training-free visual token compression reduces this cost, but performance can degrade substantially under aggressive budgets. We identify two related problems: mis-selection, where the limited visual budget is spent on answer-irrelevant frames, and discarded-context, where unselected frames retain substantial query-aligned visual signal but are removed entirely. We propose ViCoS (Verify-then-Correct with Context Mosaic), a training-free framework that addresses both problems while keeping the final answer-stage visual-token budget fixed.VerCoRe self-verifies the current retrieved frames and, when the evidence is insufficient, generates corrective visual concepts to refine CLIP retrieval. CoMo preserves discarded temporal context by packing the remaining candidate frames into a single low-resolution mosaic. Under a budget of K images, ViCoS uses K-1 full-resolution focus frames and one mosaic; at K=1, it reduces to CoMo alone. Across two backbones and five video QA benchmarks, ViCoS achieves the highest compressed accuracy among the compared methods in all 30 evaluated backbone–benchmark–budget settings. On VideoVista with InternVL3-8B, ViCoS achieves 78.80 accuracy at a 12.5% visual-token budget, only 0.04 points below the full K=8 reference while using a single encoder image.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Understanding, Video Token Compression, Vision Language Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_XXLVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ViCoS, a training-free method for video qa with limited visual input. ViCoS uses a component called VerCoRe, which allows the VLM to examine the currently retrieved frames. If the evidence is not enough, it will generate new concepts to guide new retrieval. Additionally, it uses a component called CoMo, which assembles unselected frames into a single image grid. The final input consists of K−1K-1 high-quality frames plus this compact image. ViCoS is evaluated on two different backbones, five video qa benchmarks, and three compression budgets.

Critical strengths
  1. ViCoS is a training-free method, and it can be applied directly to any VLMs. The whole test-time-scaling style pipeline is also clear and reasonable.

  2. Table 2-3 in the paper distinguishes the specific contributions of two component and their combination to ViCoS. Table 4 compares various retrieval strategies. Table 5 shows the latency comparison between ViCoS and some closely related works. These ablation studies further show the paper's specific contributions.

Critical weaknesses
  1. Although this paper claims ViCoS reduces the visual token budget, but it is only for the final input to VLMs. Actually, in the whole ViCoS pipeline, it processes more frames and visual tokens than the uniform baseline. For example, for K=2, based on Table 5 ViCoS uses 1.4 verifier calls, and it is 8.4 frame inputs. This is reflected in the latency (9.68s is double the uniform baseline of 4.88s, also is the slowest one). Also, the latency of uniform-K=8 (which as the baseline in Table 1) is missing now. This is important cause on many benchmarks the performance of uniform-K=8 is higher than or similar to ViCoS-K=2. Without the comparison on the latency, I am concerned if ViCoS actually provides a better accuracy-efficiency balance.

  2. ViCoS contains two components (VerCoRe and CoMo), but based on Table 2, VerCoRe seems not to help alone. Adding VerCoRe alone even has some degradation at K=2. But combining VerCoRe and CoMo brings huge improvements. This is an interesting result but there is no detailed analysis of this interaction. It is important for me because if we combine the results on MVBench from Table 1-3, adding VerCoRe or CoMo alone only improves by 2 points, but combining them brings 14 points improvement. This is worth adding a detailed analysis. Otherwise, the contribution of each component is still not clear.

  3. This point is related to the previous one. The idea of each component is already explored in VLMs. For example, VideoAgent [1] already used iterative retrieval with CLIP and LLM-based verification as VerCoRe. IG-VLM [2] and Video Panels [3] already used image grids to compact multiple frames into one image. Unless the authors provide a convincing explanation of what is new in combining these two components and why their combination can lead to much larger improvements than using either component alone, I do not think the novelty and contribution of this paper are enough.

  4. Table 4 shows that verification only improves the performance by 0.31 and 0.09 over content-word retrieval on Qwen2.5-VL-7B on VideoMME, and the verification requires extra VLM calls. I am concerned if the additional VLM verification is really necessary.

[1] Wang et al. VideoAgent: Long-form Video Understanding with Large Language Model as Agent. ECCV 2024.

[2] Kim et al. An Image Grid Can Be Worth a Video: Zero-shot Video Question Answering Using a VLM. arXiv 2024.

[3] Doorenbos et al. Video Panels for Long Video Understanding. CVPR 2026.

Questions for authors
  1. Could you provide the end-to-end latency of Uniform-K=8 in Table 5 and compare it with ViCoS-K=2?

  2. Could you explain why combining VerCoRe and CoMo brings much larger improvements than using any component alone, especially on MVBench?

  3. Could you clarify the novelty of ViCoS compared with VideoAgent [1], IG-VLM [2], and Video Panels [3]?

  4. Could you provide the same verification ablation as Table 4 on InternVL3-8B, especially on MVBench, to show how much the additional VLM verification contributes?

Reviewer_wViEOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper aims to address two problems in long video understanding that affect models performance under 'agressive budgets'. The authors identify the problem as 'mis-selection' and 'discarded-context', demonstrate their existence through experiments, and propose Verify-then-Correct Retrieval (VerCoR) and Context Mosaic (CoMo) to tackle them respectively. VerCoR is a self-verifying and improving loop that refines the selected keyframes. CoMo gathers the high-valued yet discarded frames into a mosaic-type image. The method is evaluated on several benchmarks under a very restrictive frame budget. Ablation study of the two proposed components is conducted.

Critical strengths
  1. The paper is well-written and easy to follow.
Critical weaknesses
  1. The motivation for the problem setup is not convincing. Handling long video input to the vision-language-model is a valid problem, but limiting the input frame budget to an extremely low number is questionable. The whole paper is generally built around the scenario where input frame numbers are limited to {8,4,2,1}. Take InternVL3-8B model adopted in the paper as example, one input frame only costs 256 tokens, which is comparable to a medium-length paragraph of text that is pretty common in use-case scenarios nowadays. Therefore, it would be preferable if the authors could move on to tackle longer videos with a larger frame budget.
  2. Similar approaches to both modules can be found in earlier papers, yet the authors provide no discussion of related work in the main text. refs: [1] VideoAgent: Long-form Video Understanding with Large Language Model as Agent, Wang et al. [2] Video Panels for Long Video Understanding, Doorenbos et al.
Questions for authors
  1. In the ablation study (Table 2), VerCoR alone does not improve from the baseline. Why is that?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.