acceptodds
Under review as a conference paper at ICLR 2027

Video Understanding at a Glance: What Remains at Two Tokens per Frame?

Abstract

Embodied and wearable systems often need a fast, low-cost understanding of ongoing visual experience before finer-grained perception or action is required. We formulate this need as *glance video understanding*: forming an overall understanding of scene context, salient events, and their temporal evolution—rather than exhaustively recovering every visual detail—at low visual cost. Preserving every frame as a distinct unit of visual evidence, we ask whether such understanding can still be sustained when the available visual-token budget is pushed to an extreme. Unlike an isolated image, a video frame carries both its *current visual state* and its *temporal contribution* relative to preceding observations. Compressing both into a single token would force these two aspects to compete within the same representational slot; we therefore set the minimum per-frame budget to two tokens—one for *what is present* and one for *what is changing*. This leads to our central question: *what remains of video understanding under this purely visual, query-independent boundary?* We investigate this question with **GlanceVLM**, which pairs an **Anchor Token** for global semantic context with an **Innovation Token** for temporally distinctive evidence. Difference-guided innovation encoding uses inter-frame changes to select informative regions while retaining their current-frame visual content, allowing both tokens to be formed before any downstream question is known. Across eight benchmarks, GlanceVLM reaches **63.6%** on MLVU and **50.0%** on our 2,400-question **GlanceVU**, showing that substantial video-understanding capability survives rather than collapses at 2 tokens/frame.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Understanding, Visual Language Model, Fixed-Budget Glance-Style Understanding

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_8RD7OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the problem of video understanding under an extremely low visual token budget, proposing GlanceVLM, a video compression framework based on dual token representation: each sampled video frame is compressed into two visual tokens that are independent of the downstream query. The Anchor Token is used to preserve the global scene semantics, while the Innovation Token selects regions with significant changes through the feature differences of adjacent frames to capture dynamic visual information. Subsequently, the two tokens interact and update through shared self-attention and respective cross-attention modules, and are input as compact video representations to the language model. Additionally, this paper proposes GlanceVU, a video understanding evaluation set containing 2,400 questions, for systematically evaluating the retained perception, temporal understanding, and reasoning capabilities under an extremely low visual budget. The experiments cover eight existing benchmarks and GlanceVU, and have verified the effectiveness of the method on multiple visual language models.

Critical strengths
  1. This paper addresses specific research questions regarding video understanding under extremely low visual token budgets and employs a query-agnostic visual encoding approach, enabling the reuse of identical video representations across diverse downstream tasks. This setup holds practical significance for resource-constrained video understanding scenarios.
  2. The authors encode global semantics and local dynamic changes as "Anchor Tokens" and "Innovation Tokens," respectively, preserving complementary information through difference-guided region selection and dual-token interaction. Compared to direct pooling or simple token pruning, this role-based compression strategy offers novelty and yields significant experimental gains.
  3. The paper presents a comprehensive evaluation across multiple video understanding benchmarks and Vision-Language Models (VLMs) of varying scales, and introduces the human-verified GlanceVU dataset. Additionally, the authors provide analyses on token ablation, capability recovery under different compression budgets, and inference latency comparisons, making the experimental findings highly valuable as a reference.
Critical weaknesses
  1. After removing the Anchor Token from GlanceVLM-8B, the average of the eight reported metrics dropped only from 54.96 to 54.79—a difference of just 0.17 percentage points—with some tasks even showing improvement. This finding somewhat conflicts with the paper's core rationale, which emphasizes the necessity of two complementary tokens per frame.
  2. The paper posits that representation competition occurs between static scene information and dynamic change information within a single token, prompting the use of two separate tokens for modeling; however, the paper has not directly verified whether such representation competition actually exists.
  3. Table 1 shows significant gaps in the results for several existing methods across various video understanding benchmarks.
Questions for authors
  1. It is recommended to include additional experiments and explanations to justify the necessity of using two complementary tokens per frame.
  2. Please provide further results for Table 1 across various video understanding benchmarks.
  3. Regarding the manual annotation for GlanceVU, could you provide more details on the specific criteria used to determine "Visual Sufficiency" and "Answer Uniqueness"?
4 comments

We thank the reviewer for recognizing the value of question-independent encoding, role-based representations, and capability-oriented evaluation. We have added two direct one-token retraining variants and have addressed the concerns below.

R3 – W1. Small Anchor-output removal effect versus complementarity

R3 – W1: After removing the Anchor Token from GlanceVLM-8B, the average of the eight reported metrics dropped only from 54.96 to 54.79—a difference of just 0.17 percentage points—with some tasks even showing improvement. This finding somewhat conflicts with the paper’s core rationale, which emphasizes the necessity of two complementary tokens per frame.

Response:

The original experiment removes an output after interaction; the new retrained architectures test whether the branch and separate states are useful.

  1. As stated beside Table 4, the ablation retains complete dual-query refinement and removes an output only immediately before LLM input. Its purpose is to compare final-readout importance under our design: Innovation removal causes −3.41 points on 8B, versus −0.17 for Anchor. This asymmetry shows that the Innovation token captures more visual information from dynamically changing regions, as intended.
  2. We have retrained models entirely without the Anchor branch and with a single shared token, and both alternatives fall below the full dual-token model. The complete 8B model losses 1.80 after removing the Anchor branch and 2.05 with a shared state. At 4B, the full model losses 2.37 and 1.97. These new results are presented together with the original output-only ablation, making the contributions of token-count and token-role design explicit alongside our visualization in Figures 10-13. We greatly appreciate your valuable and expert suggestions. | Scale / setting | AP | OCR | TR | S-Videomme | M-Videomme | L-Videomme | Total-Videomme | | --- | --- | --- | --- | --- | --- | --- | --- | | 4B full | 65.30 | 51.80 | 37.30 | 60.60 | 54.90 | 48.30 | 54.60 | | 4B Anchor branch removed; retrained | 56.31 | 50.36 | 36.72 | 58.44 | 53.00 | 48.33 | 53.26 | | 4B shared one-token state; retrained | 62.61 | 50.36 | 37.85 | 58.67 | 53.22 | 47.11 | 53.00 | | 8B full | 65.30 | 54.00 | 51.40 | 64.40 | 58.70 | 51.60 | 58.20 | | 8B Anchor branch removed; retrained | 63.96 | 50.36 | 48.59 | 63.20 | 55.78 | 49.44 | 56.14 | | 8B shared one-token state; retrained | 57.66 | 48.20 | 49.72 | 61.00 | 56.11 | 48.78 | 55.30 |

R3 – W2. Evidence for representation competition

R3 – W2 & Q1: The paper posits that representation competition occurs between static scene information and dynamic change information within a single token; however, the paper has not directly verified whether such representation competition actually exists.

Response:

We have added the shared-token experiment to test information-allocation trade-offs, while the role diagnostics show how the learned states differ.

  1. The new shared-token ablation directly tests the capacity-allocation rationale: combining the two roles into one state produces selective losses, while separate states yield higher overall accuracy. Here “representation competition” refers to asking one token to preserve both global context and change-sensitive evidence. With one shared token, mean accuracy falls by 1.97 points at 4B and 2.05 at 8B under the same training recipe.
  2. The task breakdown on VideoMME shows consistent gains of our full model in visual detail and spatial reasoning at both scales. Relative to the shared-token control, role separation improves Attribute Perception by 2.69 / 7.64 points, Object Recognition by 3.38 / 5.98, and Spatial Reasoning by 3.54 / 3.60 at 4B / 8B. At 8B, it also improves Temporal Perception by 7.32 points and Action Recognition by 5.41 points.

R3 – W3. Gaps and missing results in Table 1

R3 – W3: Table 1 shows significant gaps in the results for several existing methods across various video understanding benchmarks.

Response:

  1. Table 1 reports GlanceVLM at the deliberately restrictive two-token boundary, whereas several literature baselines use much larger visual budgets. For example, Kangaroo-8B uses 16,384 visual positions, versus our 451.78 (36.3× fewer), while GlanceVLM-8B achieves 58.2 vs. 56.0% on VideoMME. To make the budget-matched evidence easier to compare, we have extracted the comparable low-token results below and added the recent same-scale evaluations below. | Method, tokens/frame | VideoMME | MLVU | EgoSchema | GlanceVU | | --- | --- | --- | --- | --- | | LLaMA-VID, 2 | 28.3 | 23.7 | 23.7 | 26.33 | | TimeChat-Online, 2 | 54.9 | 62.4 | 39.7 | 45.92 | | GlanceVLM-4B, 2 | 54.6 | 63.0 | 48.5 | 48.38 | | GlanceVLM-8B, 2 | 58.2 | 63.6 | 52.9 | 50.00 |
Added baseline,tokens/frameVideoMME-totalsml
VideoChat3-4B, 246.3046.4449.0043.44
LLaVA-OneVision-2-8B, 243.4844.7843.0042.67
InternVL3.5-8B, 248.2250.5648.4445.67
GlanceVLM-4B, 254.6060.6054.9048.30
GlanceVLM-8B, 258.2064.4058.7051.60

Under the fair comparison conditions described above, our model significantly outperforms other models of the same size. In fact, even our 4B model surpasses 8B models by more than 5%.

3 – Q2. Further results for Table 1

R3 – Q2: Please provide further results for Table 1 across various video understanding benchmarks.

Response:

  1. We have expanded the comparison with additional recent 4B/8B baselines and additional lvbench results for GlanceVLM. We adds VideoChat3-4B, LLaVA-OneVision-2-8B, and InternVL3.5-8B on VideoMME. The results originally shown in Table 1 were intended to directly characterize the model’s capabilities under the central setting of interest; as Figure 3 in the submitted manuscript shows, our model’s capabilities can be further improved. | Scale / setting | MLVU | Δ | VideoMME | Δ | EgoSchema | Δ | LVBench | Δ | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 4B, 2 tokens/frame | 63.04 | — | 54.60 | — | 48.52 | — | 38.50 | — | | 4B, r=0.2 | 64.03 | +0.99 | 58.07 | +3.47 | 50.33 | +1.81 | 43.12 | +4.62 | | 8B, 2 tokens/frame | 63.58 | — | 58.20 | — | 52.87 | — | 37.70 | — | | 8B, r=0.2 | 65.82 | +2.24 | 62.48 | +4.28 | 55.20 | +2.33 | 43.54 | +5.84 |

R3 – Q3. Visual Sufficiency and Answer Uniqueness

R3 – Q3: Regarding the manual annotation for GlanceVU, could you provide more details on the specific criteria used to determine “Visual Sufficiency” and “Answer Uniqueness”?

Response:

  1. “Visual Sufficiency” means that the evidence needed to answer is observable from the video frame alone, without requiring audio or subtitle information.
  2. “Answer Uniqueness” means that exactly one provided option is supported at the question’s intended scope. Annotators check the correct answer against the video and verify that another option is not equally defensible because of synonymous wording, overlapping meanings, mismatched granularity, or ambiguous references. An item is rejected if two or more options appear equally plausible.
  3. The key criterion combining these two requirements is therefore one visually justified answer. We have made the operational definitions explicit in Appendix A.1

I thank the authors for the rebuttal. After carefully reading all reviews and the rebuttal, I would keep my initial judgment (weak accept), as the rebuttal addressed my main concerns.

Thank you very much for carefully reading our rebuttal. We are glad that our responses have addressed your concerns, and we appreciate your feedback on our reply.

Reviewer_JPjxOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces two tokens per frame for video understanding

  • Anchor Token for global semantics
  • Innovation Token for temporal info Starting from the normal regular vision encoding patches, though pulling, self-attention, cross-attention, these patches will form the two tokens per frame above.

A video will be encoded in this two tokens per frame way and be fed into a llm, named GlanceVLM, designed for video understanding tasks.

The authors evaluated this method on several benchmarks like EgoSchema, LongVideoBench, and also introduced a new bench GlanceVU for extreme compression cases.

Critical strengths
  • The two-token design shows its effectiveness of the compression by comparing to other compression methods like LLaMA-VID on a good range of video eval benchmarks.
  • A model filtered and human verified benchmark, GlanceVU, enabled a more systematically analysis of fine-grained categories for video tasks.
Critical weaknesses
  • More ablations and apple-to-apple comparisons are needed to prove the current hand-crafted two-token design.
    • 1\ model size is a confounder
    • 2\ what if without hand-crafted heuristics just learn a compressor to compose one frame to two tokens
      • it's essentially a hand-crafted spatial and temporal compression where for the temporal the sliding window size is 2.
    • 3\ the contribution of the 2 tokens are unclear, making it questionable of the necessity of the design like this.
      • for instance, in table 4, token ablations, at 8B removing the anchor token only -0.17 of the performance.
  • Missing major previous work
Questions for authors
  • Maybe a diagram of x-axis token cost, y-axis performance, circle size model size could help a bit for visualizing where this work is positioned from the literature
5 comments

R2 – W1. Model size as a confounder

R2 – W1: Model size is a confounder. More ablations and apple-to-apple comparisons are needed to prove the current hand-crafted two-token design.

Response:

Tables 1, 3, and 6 of our submitted manuscript already report experiments across 4B/7B/8B model sizes under matched experimental conditions, demonstrating the effectiveness of our method at multiple scales, particularly in Tables 3 and 6. Tables 3 and 6 further test the 8B model under three different settings to establish that the performance gains arise from our two-visual-token design, providing evidence for GlanceVLM through both cross-model comparisons and within-model ablations.

  1. At the identical 2-token/frame method and matched training configuration, GlanceVLM improves VideoMME by +5.5 / +8.1 / +12.3 points on Qwen2.5-VL-7B, Qwen3-VL-4B, and Qwen3-VL-8B, respectively (Tables 3 and 6(a)); gains occur in every duration range. These paired experiments isolate the benefit beyond simply choosing a larger pretrained model. Table 3 additionally shows improvements in subtask–backbone pairs.
  2. Table 1 compares the latest third-generation Qwen-VL baselines at 4B and 8B with similarly sized models across multiple tasks. Under a unified re-evaluation protocol, these comparison models achieve lower or similar accuracy despite using more tokens than ours. This is consistent with our analysis of Table 1: “Competitive long-video accuracy with only two visual tokens per frame.”
  3. We sincerely thank you for the plotting suggestion. To make the same-scale comparisons immediately apparent, we have added a single cost–accuracy–model-size plot that brings together comparison results previously distributed across multiple tables by experimental setting. More importantly, we have added comparisons with recent 4B/8B models under the same fair two-token budget. The table below reports VideoChat3-4B, LLaVA-OneVision-2-8B, and InternVL3.5-8B under the recorded low-budget settings. GlanceVLM-8B attains the highest accuracy among the plotted configurations (58.2%) at 451.78 visual positions. We have also incorporated these additional experimental results into the suggested plot, making our model’s capabilities clearer.

Additional same-scale evaluations on VideoMME as follows:

ModelScaleFrame captokens/frameShortMediumLongTotal
VideoChat34B768246.4449.0043.4446.30
GlanceVLM4B768260.6054.9048.3054.60
LLaVA-OneVision-28B256244.7843.0042.6743.48
LLaVA-OneVision-28B7682---out of memory on a single A100 80G
InternVL3.58B768250.5648.4445.6748.22
GlanceVLM8B768264.4058.7051.6058.20

R2 – W2. Learning compression without heuristics; temporal “window size 2”

R2 – W2: What if without hand-crafted heuristics just learn a compressor to compose one frame to two tokens? It is essentially a hand-crafted spatial and temporal compression where for the temporal the sliding window size is 2.

Response:

First, we would like to clarify that our method is not equivalent to “spatial and temporal compression where for the temporal the sliding window size is 2”; this characterization overlooks the important operation of using the video itself to guide evidence selection. Our method provides a distinctive modeling approach that does not rely on external guidance, seeks to preserve information from every video frame, and highlights key dynamic visual changes that help advance the narrative. It therefore cannot be equated with a conventional two-frame compressor. As summarized in the method overview, the distinction between “where to read” and “what to retain” cannot be achieved by direct compression with a sliding window of two frames. Second, the appendix of our submitted manuscript already reports experimental results and analysis for module-free, end-to-end learning under a two-token-per-frame compression interface. These results further substantiate the advantages of our specific design.

  1. The requested “remove the specialized module and learn under a two-token interface” experiment is already included: it achieves 53.4%, versus 58.2% for GlanceVLM-8B. In Ours-8B-Pool (Tables 6(a) and 3), the Anchor–Innovation module is removed, each frame is reduced to two positions through standard pooling-based compression, and the entire VLM is trained end-to-end using the same data and schedule. The +4.8-point gain therefore directly demonstrates the benefit of our structured representation over the module-free, end-to-end-trained alternative.
  2. The subtask gains show what the specialized design adds beyond generic bottleneck adaptation. Relative to that control, GlanceVLM-8B improves Temporal Perception 52.7→65.5 (+12.8), Temporal Reasoning 27.7→51.4 (+23.7), Object Recognition 54.0→60.5 (+6.5), and OCR 46.8→54.0 (+7.2). These changes are consistent with preserving global appearance and temporally distinctive evidence through separate roles. Moreover, the queries and their aggregation layers are learned end-to-end; the residual-based region selection specifies where to read, not a hand-written answer representation.
  3. Our use of adjacent frames is an evidence-selection operation, not two-frame temporal aggregation. A two-frame window compressor combines the content of both frames into a shared window representation. In GlanceVLM, the previous frame only helps locate changed regions; both output tokens read current-frame evidence, and every frame keeps its own pair. The difference is illustrated below:
Two-frame compressor (window = 2)
+---------+   +---------+
|  X(t-1) |   |  X(t)   |
+----+----+   +----+----+
+-----------v-----------+
       merge BOTH  ------> window tokens

GlanceVLM (selection)
+---------+   +---------+
|  X(t-1) |   |  X(t)   |
+----+----+   +----+----+
     +-- difference --+       Where to read
                  |
              +----v----+
              |  X(t)   |      What to retain
              | [select region in X(t)]|
              +----+----+
                  +---------> 2 token

We consolidate our results in Table 6&4 below:

8B settingSMLVideoMME-Total
Module removed; end-to-end trained (Ours-8B-Pool)63.049.347.953.40
Text-only44.042.044.743.60
Anchor branch removed; retrained63.2055.7849.4456.14
Shared one-token; retrained61.0056.1148.7855.30
Full GlanceVLM-8B64.4058.7051.6058.20

R2 – W3: The contribution of the 2 tokens are unclear, making it questionable of the necessity of the design like this. For instance, in Table 4, token ablations, at 8B removing the anchor token only −0.17 of the performance.

Response:

The small 0.17-point difference, contrasted with the 3.41-point difference, was intended to demonstrate that, after our role-specialized training, the Innovation token achieves our design objective of capturing key information within video frames. Retraining without that branch produces larger losses. As indicated by the ablation subsection title, the purpose of the 0.17-point experiment is “Disentangling token roles.” To examine whether one or two tokens should be used, we have designed new settings that directly test the token-count choice.

  1. The −0.17 result is a final-output ablation after dual-query interaction. As the text to the left of Table 4, it explicitly retains the complete refinement computation and deletes one output only before LLM input. This was designed to identify the stronger final readout: removing Innovation costs −3.41, compared with −0.17 for Anchor on 8B.
  2. The new architectural ablations remove that ambiguity: both retrained one-token alternatives lose accuracy at both scales. The 8B Anchor-removed variants show a 1.80point drop in mean accuracy relative to the full model; the corresponding 4B losses are −2.37.
  3. We have designed a variant that uses only one shared token from the outset to perform the modeling originally assigned to two tokens. The new one-token models show a 2.05-point drop in mean accuracy and a 2.9-point drop in VideoMME Total. This futher supports the actual 2 token design rationale.

R2 – W4. Missing VoCo-LLaMA and ReWind

R2 – W4: Missing major previous work: (1) VoCo-LLaMA, CVPR 2025; (2) ReWind, CVPR 2025.

Response:

VoCo-LLaMA and ReWind are relevant works, but their focus differs from that of our model. We have added both works to Related Work and connected their efficiency goals to our study of glance video-understanding capability on GlanceVU.

  1. These are relevant works, but their compression mechanisms and complete methods differ from GlanceVLM’s fixed visual-side dual-role representation. VoCo-LLaMA (CVPR 2025) uses LLM-internal attention and distillation to compress visual information into VoCo activations/KV cache, whereas we generate Anchor–Innovation tokens before the LLM through a dedicated visual module. Its LLM-facing visual cost is difficult to delineate under our definition, although its implementation is complementary to ours. ReWind (CVPR 2025) uses an instruction-aware perceiver; it uses instruction text to select dense-frame evidence, whereas GlanceVLM fixes its visual representation before the question is known.
  2. We have added both works to Related Work. Since ReWind has released neither code nor model weights, whereas VoCo-LLaMA has released only code, we have reproduced VoCo-LLaMA. Its video accuracy is substantially below that of both GlanceVLM-4B and GlanceVLM-8B. Given that its architecture primarily targets image representations, this comparison demonstrates the advantage of our method for video understanding. | Method | VideoMME | S | M | L | | --- | --- | --- | --- | --- | | VoCo-LLaMA reproduction | 9.63% | 24.56% | 2.56% | 1.78% | | LLaMA-VID | 28.3% | 31.8% | 27.0% | 26.2% | | GlanceVLM | 58.20% | 64.4% | 58.7% | 51.6% |

R2 – Q1. Token-cost / performance / model-size plot

R2 – Q1: Maybe a diagram of x-axis token cost, y-axis performance, circle size model size could help a bit for visualizing where this work is positioned from the literature.

Response:

Thank you very much for the plotting suggestion. We have incorporated the plot and the additional results discussed above into the revised manuscript. We sincerely hope that the above responses help to clarify some of the misunderstandings that may have arisen.

Reviewer_RAViOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents an interesting question-independent video understanding view. The paper asks a useful question: how much video understanding can be retained when the language model receives only two visual tokens per frame unit? The GlanceVLM uses one token for global appearance and another for current-frame regions selected through temporal differences which aligns with the intuition and supported by cross-backbone evaluation and relevant same-backbone controls.

Critical strengths

The question-independent setting is well motivated. The visual representation can be computed once and reused for different questions. The paper also makes a useful distinction between using temporal differences to select regions and using those differences as the visual content itself.

The evaluation includes multiple backbones, pooling and sparse-frame controls, runtime measurements, and failure cases. GlanceVU’s construction is well-documented.

Critical weaknesses

W1. Training need more clarification

Paper should state whether they use the same training data, optimization budget, initialization, and trainable parameters under same-backbone setting, while I currently assuming they are, correct me if i was wrong.

Comparison under matched training would be more informative.

W2. Minimum-token argument is unsupported

Removing the final Anchor output with only small performance change weakens the claim that two tokens design are necessary.

However, both queries have already participated in refinement, so this experiment does not establish that the internal Anchor computation is redundant. Just need more proof to support the claim that two tokens presented as a design choice.

W3. Training–evaluation overlap

Paper combines existing benchmark sources, while training uses Video-R1-Data. The paper is suggested to provide source splits and checks for repeated videos, questions, and close duplicates.

W4. Compression and efficiency

The paper demonstrates a large reduction in LLM-facing visual positions. IBut ts broader efficiency interpretation is less clear because dense encoding and other visual processing remain. Table 6 already contains timing results, but the units and timing boundaries need to be specified.

Questions for authors

Q1. Adjacent frame temporal differences may capture more prominent camera motion rather than task-relevent changes. I would like to see the reply of this concern.

Q2. What accounts for the reduction from 31,275 to 27,808 examples?

Q3. Could you state more precisely what GlanceVLM adds beyond LLaMA-VID’s two-token representation and TimeChat-Online’s question-independent temporal filtering?

Q4. Could you discuss the relationship between your learned-query compressor and prior architectures such as BLIP-2, clarifying which components are inherited and which are specific to your method?

5 comments

We thank the reviewer for recognizing the question-independent setting, the distinction between change-guided selection and visual content, and the breadth of the evaluation. We have addressed the concerns below.

R1 – W1: Paper should state whether they use the same training data, optimization budget, initialization, and trainable parameters under same-backbone setting. Comparison under matched training would be more informative.

Response:

  1. Yes—we use the same training data, backbone initialization, optimization schedule, and end-to-end fine-tuning protocol. As specified in §4.1 (Implementation details), all are trained in one SFT stage on the same video-only examples, with all corresponding backbone components trainable, a per-device batch size of 4, and the same learning rates.

R1 – W2: Removing the final Anchor output with only small performance change weakens the claim that two tokens design are necessary. However, both queries have already participated in refinement, so this experiment does not establish that the internal Anchor computation is redundant.

Response:

We sincerely thank you for your attention to our experimental design and for your suggestions regarding further experiments. We have retrained one-token models to provide a more direct test.

  1. As you correctly noted, the −0.17 result was not intended to establish whether the architecture should use one token or two. Rather, it was designed to test whether, after our deliberately role-specialized training, the Innovation token learns to capture more information that is useful for the video narrative. This comparison verifies that the Innovation token achieves our original design objective of capturing key information within video frames. This protocol is explicitly stated beside Table 4 in our submitted paper: both tokens undergo the complete refinement process, and one output is removed immediately before LLM input. The surviving Innovation state can therefore retain context supplied by Anchor.
  2. We have directly tested the architectural alternative by retraining two one-token variants to more directly ablate the token-count design. The first completely removes the Anchor token and its associated computation and is retrained and evaluated; the second uses one shared token to perform the original two-role computation and is also retrained. On 8B, their eight-score means are 53.16/52.91, versus 54.96 for the full model (−1.80/−2.05). On 4B, they reach 50.52/50.92, versus 52.89 (−2.37/−1.97).
  3. The combined evidence supports two role-specialized tokens as an effective design choice, with gains from both state separation and evidence selection. We have incorporated the results of these two valuable experiments suggested by you into Table 4 and the ablation analysis in the revised manuscript: | Scale | Setting | LLM tokens/frame | VMME S | M | L | Total | MLVU | EgoSchema | OVO Real | OVO Back | Mean | Δ | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 4B | Full dual-token model | 2 | 60.60 | 54.90 | 48.30 | 54.60 | 63.04 | 48.52 | 51.87 | 41.29 | 52.89 | — | | 4B | Final Anchor output removed; no retraining | 1 | 59.20 | 54.20 | 47.60 | 53.70 | 59.75 | 47.51 | 51.18 | 38.44 | 51.45 | −1.44 | | 4B | Anchor branch removed; retrained | 1 | 58.44 | 53.00 | 48.33 | 53.26 | 58.41 | 45.10 | 50.40 | 37.23 | 50.52 | −2.37 | | 4B | Shared one-token state; retrained | 1 | 58.67 | 53.22 | 47.11 | 53.00 | 61.14 | 46.87 | 49.30 | 38.09 | 50.92 | −1.97 | | 8B | Full dual-token model | 2 | 64.40 | 58.70 | 51.60 | 58.20 | 63.58 | 52.87 | 52.44 | 37.91 | 54.96 | — | | 8B | Final Anchor output removed; no retraining | 1 | 64.40 | 58.00 | 50.10 | 57.50 | 61.73 | 52.95 | 52.64 | 41.03 | 54.79 | −0.17 | | 8B | Anchor branch removed; retrained | 1 | 63.20 | 55.78 | 49.44 | 56.14 | 60.26 | 50.25 | 50.69 | 39.50 | 53.16 | −1.80 | | 8B | Shared one-token state; retrained | 1 | 61.00 | 56.11 | 48.78 | 55.30 | 60.74 | 51.25 | 50.21 | 39.90 | 52.91 | −2.05 |

R1 – W3: Paper combines existing benchmark sources, while training uses Video-R1-Data. The paper is suggested to provide source splits and checks for repeated videos, questions, and close duplicates.

Response:

The direct JSON comparison found zero exact video–question pair overlaps; source splits and video provenance are distinguished explicitly.

  1. Our direct comparison of the training and evaluation JSON files found zero exact video–question pair overlaps; although the same source dataset names may appear, the data are strictly partitioned into train/val/test subsets in fact. Video-R1-Data aggregates particular source splits, whereas evaluation uses the corresponding held-out items. For example, NExT-QA evaluation uses the multiple-choice VQA-pair split specifically designated for evaluation, which is separate from its train splits. Beyond checking these source definitions, we compared the actual training and evaluation records in their JSON files and obtained 0 matching video–question pairs. We have emphasized the absence of overlap in the corresponding paragraph.

R1 – W4: The paper demonstrates a large reduction in LLM-facing visual positions. Its broader efficiency interpretation is less clear because dense encoding and other visual processing remain. Table 6 already contains timing results, but the units and timing boundaries need to be specified.

Response:

We have clarified that Lat. is in seconds, measured from the start of visual-encoder execution after Decord video reading to the final generated output in Table 6’s caption. The units and timing boundaries follow the conventional evaluation protocol. Thank you for pointing out the need for clarification. VideoMME visual positions fall from 40,629.35 to 451.78 (89.93×); GlanceVU falls from 37,946.99 to 412.51 (91.99×). At nearly equal retained budgets, the reported model-path latency is 7.62 vs. 10.52 s on VideoMME and 6.34 vs. 7.16 s on GlanceVU, compared with TimeChat-Online. Taking into account the units and timing ranges clarified above, this demonstrates the improved efficiency of our method.


R1 – Q1. Could camera motion dominate adjacent-frame differences?

R1 – Q1: Adjacent frame temporal differences may capture more prominent camera motion rather than task-relevant changes. I would like to see the reply of this concern.

Response:

Our method does not directly select isolated visual patch tokens based on their individual changes. Instead, it selects tokens from square regions that exhibit substantial overall change. A region is therefore selected only when the change is substantial across the region as a whole, reducing interference from individual tokens and helping mitigate the effects of camera motion or jitter that you mentioned.

  1. Our method uses temporal differences to select coherent current-frame regions, not to encode camera-motion residuals as visual content. Differences are computed in feature space and used to select regions rather than isolated high-response patches (§3.2). The Innovation token then aggregates the actual current-frame features in those regions; Anchor independently retains full-frame context. The design therefore favors coherent regional change while preserving appearance evidence, instead of treating every large residual as task-relevant motion.
  2. The submitted ablations and egocentric evaluation show that this lightweight selection yields useful evidence in practice. Replacing the selected regional evidence with full-frame evidence reduces the aggregate by 5.01/3.35 points on 4B/8B. On motion-rich EgoSchema, our 4B model achieves 48.52%, versus 23.73% for LLaMA-VID-7B, while our 8B model reaches 52.87%. Figs. 9–12 also show selected regions around pose changes, hand–object interactions, and object-state transitions.
  3. Global-motion alignment or flow-based routing can replace the lightweight residual estimator, but it introduces additional computation. The present method deliberately uses a low-cost visual cue and learns how to aggregate the resulting current-frame evidence while mitigating the effects of camera motion to some extent.

R1 – Q2. Why 31,275 → 27,808 examples?

R1 – Q2: What accounts for the reduction from 31,275 to 27,808 examples?

Response:

  1. As described in Appendix A.1, 31,275 questions are reduced to 27,808 candidates remaining after source-level release constraints: some videos from upstream datasets are subject to additional, somewhat stricter usage agreements, so we excluded them from the candidate pool to avoid potential disputes. These 27,808 candidates are then further filtered through the Semantic Audit process to obtain the final 2,400-question benchmark.

R1 – Q3. Beyond LLaMA-VID and TimeChat-Online

R1 – Q3: Could you state more precisely what GlanceVLM adds beyond LLaMA-VID’s two-token representation and TimeChat-Online’s question-independent temporal filtering?

Response:

Our goal is to determine whether a model can grasp the general content of an input video at a glance, as humans do, from the video itself and without considering downstream questions. More specifically, we ask whether video understanding can be achieved at an extremely low token cost. Motivated by this question, we impose a modeling constraint that the method must not omit or deliberately discard the visual information represented by any sampled frame or token, and must not take shortcuts through external guidance. Accordingly, in terms of architecture and methodology, we provide a distinctive modeling approach that does not rely on external guidance, seeks to preserve information from every video frame, and highlights key dynamic visual changes that help advance the narrative. Neither LLaMA-VID, which treats each frame as an isolated image and requires question guidance to retrieve image regions, nor TimeChat-Online, which deliberately discards most visual tokens from each frame for efficiency, fulfills this objective. Thus, as stated in our Introduction, we deliberately constrain the model to the most stringent setting, providing a starting point for future work on this task.

  1. GlanceVLM studies question-independent video understanding at a fixed per-frame budget, rather than only reducing token count for a known question. Each observed frame contributes two learned, role-specialized tokens: a full-frame Anchor and a change-guided Innovation. This preserves a per-frame evidence interface and emphasizes visually meaningful evolution before any downstream question is available. The contribution is this integrated representation and the capability boundary it enables us to study, not only the number 2.
  2. Unlike LLaMA-VID’s instruction text-conditioned context token, our Innovation token introduces temporal evidence through the video itself. By treating each frame as an isolated image, LLaMA-VID’s modeling approach cannot capture the spatiotemporal relationships within a video. Our model is designed from the outset to model both temporal and spatial information, relying solely on the video itself. Thus, compared with LLaMA-VID, which relies on external text for retrieval, our method provides a way to effectively extract information from a video rather than processing it as isolated images. This is a major modeling distinction between our approach and the image-based LLaMA-VID approach. The need to model spatiotemporal structure also explains why strong performance in previous image-understanding or short-video question-answering models does not transfer seamlessly to long videos.
  3. Unlike TimeChat-Online’s hard dropping of original patches, GlanceVLM learns two complementary summaries with persistent global access. TimeChat-Online already performs question-independent adjacent-frame redundancy filtering; our distinction is aggregating all information in a video rather than dropping/deletion. Thus, relative to our submitted two-token setting, to reach approximately our VideoMME score, its threshold-based setting uses 1,501.85 tokens versus our 451.78, or 3.32× the visual-position cost, to recover the discarded information.

R1 – Q4: Could you discuss the relationship between your learned-query compressor and prior architectures such as BLIP-2, clarifying which components are inherited and which are specific to your method?

Response:

The new design in our model is role-dependent initialization, asymmetric visual evidence, and two role-specific forward computation branches, without any pretrained Q-Former weights. The only inherited element is the basic idea of reading visual features through learnable queries and attention.

  1. We adopt the learned-query attention idea, not the original Q-Former architecture or pretrained BLIP-2/InstructBLIP Q-Former weights; our compressor is trained from scratch. Query self-attention and visual cross-attention are established building blocks. We have added the BLIP-2 citation to acknowledge the inspiration from this learned-query attention idea.
  2. Our specific design is a video-specific two-role information extractor, rather than a generic set of interchangeable visual queries. Anchor is initialized from a full-frame mean and reads all frame patches; Innovation is initialized from the max-pooled change-selected support and reads its regional descriptors. Their independently parameterized cross-attention branches preserve these distinct evidence sources while shared query self-attention exchanges context (§3.1–3.3). Thus, our novelty lies in role initialization, visual-only evidence routing, and coupled refinement **under the two-token video constraint.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.