- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses the problem of video understanding under an extremely low visual token budget, proposing GlanceVLM, a video compression framework based on dual token representation: each sampled video frame is compressed into two visual tokens that are independent of the downstream query. The Anchor Token is used to preserve the global scene semantics, while the Innovation Token selects regions with significant changes through the feature differences of adjacent frames to capture dynamic visual information. Subsequently, the two tokens interact and update through shared self-attention and respective cross-attention modules, and are input as compact video representations to the language model. Additionally, this paper proposes GlanceVU, a video understanding evaluation set containing 2,400 questions, for systematically evaluating the retained perception, temporal understanding, and reasoning capabilities under an extremely low visual budget. The experiments cover eight existing benchmarks and GlanceVU, and have verified the effectiveness of the method on multiple visual language models.
Critical strengths
- This paper addresses specific research questions regarding video understanding under extremely low visual token budgets and employs a query-agnostic visual encoding approach, enabling the reuse of identical video representations across diverse downstream tasks. This setup holds practical significance for resource-constrained video understanding scenarios.
- The authors encode global semantics and local dynamic changes as "Anchor Tokens" and "Innovation Tokens," respectively, preserving complementary information through difference-guided region selection and dual-token interaction. Compared to direct pooling or simple token pruning, this role-based compression strategy offers novelty and yields significant experimental gains.
- The paper presents a comprehensive evaluation across multiple video understanding benchmarks and Vision-Language Models (VLMs) of varying scales, and introduces the human-verified GlanceVU dataset. Additionally, the authors provide analyses on token ablation, capability recovery under different compression budgets, and inference latency comparisons, making the experimental findings highly valuable as a reference.
Critical weaknesses
- After removing the Anchor Token from GlanceVLM-8B, the average of the eight reported metrics dropped only from 54.96 to 54.79—a difference of just 0.17 percentage points—with some tasks even showing improvement. This finding somewhat conflicts with the paper's core rationale, which emphasizes the necessity of two complementary tokens per frame.
- The paper posits that representation competition occurs between static scene information and dynamic change information within a single token, prompting the use of two separate tokens for modeling; however, the paper has not directly verified whether such representation competition actually exists.
- Table 1 shows significant gaps in the results for several existing methods across various video understanding benchmarks.
Questions for authors
- It is recommended to include additional experiments and explanations to justify the necessity of using two complementary tokens per frame.
- Please provide further results for Table 1 across various video understanding benchmarks.
- Regarding the manual annotation for GlanceVU, could you provide more details on the specific criteria used to determine "Visual Sufficiency" and "Answer Uniqueness"?
4 comments
R3 – W3. Gaps and missing results in Table 1
R3 – W3: Table 1 shows significant gaps in the results for several existing methods across various video understanding benchmarks.
Response:
- Table 1 reports GlanceVLM at the deliberately restrictive two-token boundary, whereas several literature baselines use much larger visual budgets. For example, Kangaroo-8B uses 16,384 visual positions, versus our 451.78 (36.3× fewer), while GlanceVLM-8B achieves 58.2 vs. 56.0% on VideoMME. To make the budget-matched evidence easier to compare, we have extracted the comparable low-token results below and added the recent same-scale evaluations below. | Method, tokens/frame | VideoMME | MLVU | EgoSchema | GlanceVU | | --- | --- | --- | --- | --- | | LLaMA-VID, 2 | 28.3 | 23.7 | 23.7 | 26.33 | | TimeChat-Online, 2 | 54.9 | 62.4 | 39.7 | 45.92 | | GlanceVLM-4B, 2 | 54.6 | 63.0 | 48.5 | 48.38 | | GlanceVLM-8B, 2 | 58.2 | 63.6 | 52.9 | 50.00 |
| Added baseline,tokens/frame | VideoMME-total | s | m | l |
|---|---|---|---|---|
| VideoChat3-4B, 2 | 46.30 | 46.44 | 49.00 | 43.44 |
| LLaVA-OneVision-2-8B, 2 | 43.48 | 44.78 | 43.00 | 42.67 |
| InternVL3.5-8B, 2 | 48.22 | 50.56 | 48.44 | 45.67 |
| GlanceVLM-4B, 2 | 54.60 | 60.60 | 54.90 | 48.30 |
| GlanceVLM-8B, 2 | 58.20 | 64.40 | 58.70 | 51.60 |
Under the fair comparison conditions described above, our model significantly outperforms other models of the same size. In fact, even our 4B model surpasses 8B models by more than 5%.
3 – Q2. Further results for Table 1
R3 – Q2: Please provide further results for Table 1 across various video understanding benchmarks.
Response:
- We have expanded the comparison with additional recent 4B/8B baselines and additional lvbench results for GlanceVLM. We adds VideoChat3-4B, LLaVA-OneVision-2-8B, and InternVL3.5-8B on VideoMME. The results originally shown in Table 1 were intended to directly characterize the model’s capabilities under the central setting of interest; as Figure 3 in the submitted manuscript shows, our model’s capabilities can be further improved. | Scale / setting | MLVU | Δ | VideoMME | Δ | EgoSchema | Δ | LVBench | Δ | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 4B, 2 tokens/frame | 63.04 | — | 54.60 | — | 48.52 | — | 38.50 | — | | 4B, r=0.2 | 64.03 | +0.99 | 58.07 | +3.47 | 50.33 | +1.81 | 43.12 | +4.62 | | 8B, 2 tokens/frame | 63.58 | — | 58.20 | — | 52.87 | — | 37.70 | — | | 8B, r=0.2 | 65.82 | +2.24 | 62.48 | +4.28 | 55.20 | +2.33 | 43.54 | +5.84 |
R3 – Q3. Visual Sufficiency and Answer Uniqueness
R3 – Q3: Regarding the manual annotation for GlanceVU, could you provide more details on the specific criteria used to determine “Visual Sufficiency” and “Answer Uniqueness”?
Response:
- “Visual Sufficiency” means that the evidence needed to answer is observable from the video frame alone, without requiring audio or subtitle information.
- “Answer Uniqueness” means that exactly one provided option is supported at the question’s intended scope. Annotators check the correct answer against the video and verify that another option is not equally defensible because of synonymous wording, overlapping meanings, mismatched granularity, or ambiguous references. An item is rejected if two or more options appear equally plausible.
- The key criterion combining these two requirements is therefore one visually justified answer. We have made the operational definitions explicit in Appendix A.1
I thank the authors for the rebuttal. After carefully reading all reviews and the rebuttal, I would keep my initial judgment (weak accept), as the rebuttal addressed my main concerns.
Thank you very much for carefully reading our rebuttal. We are glad that our responses have addressed your concerns, and we appreciate your feedback on our reply.
We thank the reviewer for recognizing the value of question-independent encoding, role-based representations, and capability-oriented evaluation. We have added two direct one-token retraining variants and have addressed the concerns below.
R3 – W1. Small Anchor-output removal effect versus complementarity
Response:
The original experiment removes an output after interaction; the new retrained architectures test whether the branch and separate states are useful.
R3 – W2. Evidence for representation competition
Response:
We have added the shared-token experiment to test information-allocation trade-offs, while the role diagnostics show how the learned states differ.