acceptodds
Under review as a conference paper at ICLR 2027

Before the Question Arrives: Causal Evidence Budgeting for Streaming Video Understanding

Abstract

Streaming video understanding is crucial for AI systems to interpret dynamic environments, yet providing accurate, real-time responses remains challenging. Models must process a growing visual history within limited context and memory budgets. Existing methods reduce this pressure through visual-token compression or key–value (KV) cache management, but they often fix what to retain at frame arrival, before considering earlier and later evidence together. Such early decisions can discard evidence needed by future questions, and later retrieval cannot recover it. To address this mismatch, we introduce StreamingRETAKE, a training-free framework for causal evidence budgeting that allocates visual-token and KV capacity over time. Visual-token capacity is directed toward informative frames to build a compact evidence pool, from which shared evidence is selected jointly after more frames arrive. Then retrieval combines evidence importance with question relevance to build the answer context. The framework bounds visual-history storage and answer contexts as the stream grows. Extensive experiments show that our proposed method achieves state-of-the-art accuracy of 75.52% and 81.80% on OVO-Bench Real-Time and StreamingBench real-Time, respectively.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Streaming Video Understanding, MLLM

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_WzZnOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes StreamingReTaKe, a training-free method for streaming video QA that extends ReTaKe/AdaReTaKe to the online setting. It has three parts: 1) A per-frame token budget driven by a novelty score, with a “budget bank” that carries unused tokens forward. 2) A recent window of full KVs, plus a long-term memory where tokens are selected jointly at the end of each episode. 3) Retrieval at question time that mixes question relevance with the stored importance scores. The authors claim memory that stays bounded regardless of stream length. They report results on OVO-Bench and StreamingBench with four backbones, along with some offline benchmarks.

Critical strengths

The method is training-free and works on four different backbones. The efficiency part is reasonably analyzed and frame deadlines are met at 2 FPS, peak memory is lower than HERMES, and the TTFT breakdown with pending frames is useful. For a practical system these aspects are important.

Critical weaknesses
  • It is not clear what the contribution is. The setup (a recent window, plus a compact long-term memory built before the question is known, plus question-time retrieval over both) is already common in streaming video work, for example [R1, R5, R6]. Most of the individual components also look close to existing methods: query-agnostic attention scoring for retention is close to HERMES, KV novelty and norm-based retention is close to [R4], dropping frames or tokens based on change from the previous frame is close to [R8] and FluxMem, KV retrieval with position re-indexing is close to ReKV, LiveVLM and StreamKV. The contribution list in the intro does not say which of these is the new idea, and I could not tell after reading it. Several relevant works are also not discussed (list below).

  • The results do not clearly support the claims: Compared with HERMES, the gains are small and not consistent. The method is +0.5 on StreamingBench with Qwen3-VL-8B. It is worse on OVO Backward and Avg with LLaVA-OV, worse on StreamingBench with Qwen2.5-VL, and worse on Video-MME.

  • The hyperparameters (B, K, W, M_KV) are different for every backbone. The LLaVA row in Table 1 uses K=64, W=6, which is not the stated reference setting. The method is also quite sensitive: W=64 loses 12.6 points and M_KV=2048 loses 11.8 points. It is not clear if the settings were tuned on the test benchmarks. There are no runs with different seeds or error bars, while many margins are under 1 point.

  • The TTFT advantage over HERMES mostly comes from HERMES having pending frames at question time. After pending frames are processed, HERMES is faster (72 ms vs 148 ms).

  • The paper also has no baseline that uses only the recent window. Real-time subsets are known to depend mostly on recent frames, so such a baseline is needed to show the long-term memory is actually helping.

Some missing references:

[R1] Shehreen Azad, Vibhav Vineet, Yogesh Singh Rawat. StreamReady: Learning What to Answer and When in Long Streaming Videos. CVPR 2026 (arXiv:2603.08620).

[R2] Vaggelis Dorovatas, Soroush Seifi, Gunshi Gupta, Rahaf Aljundi. Recurrent Attention-based Token Selection for Efficient Streaming Video-LLMs. NeurIPS 2025.

[R3] Azad et. al. HierarQ: Task-Aware Hierarchical Q-Former for Enhanced Video Understanding. CVPR 2025 (arXiv:2503.08585).

[R4] Minsoo Kim, Kyuhong Shim, Jungwook Choi, Simyung Chang. InfiniPot-V: Memory-Constrained KV Cache Compression for Streaming Video Understanding. NeurIPS 2025 (arXiv:2506.15745).

[R5] Haoji Zhang, Yiqin Wang, Yansong Tang, et al. Flash-VStream: Efficient Real-Time Understanding for Long Video Streams. ICCV 2025.

[R6] Zhenpeng Huang, Xinhao Li, Jiaqi Li, et al. Online Video Understanding: OVBench and VideoChat-Online. CVPR 2025.

[R7] Ruyi Xu, Guangxuan Xiao, Yukang Chen, et al. StreamingVLM: Real-Time Understanding for Infinite Video Streams. ICLR 2026 (arXiv:2510.09608).

[R8] Linli Yao, Yicheng Li, Yuancheng Wei, et al. TimeChat-Online: 80% Visual Tokens are Naturally Redundant in Streaming Videos. ACM MM 2025.

[R9] Haibo Wang, Bo Feng, Zhengfeng Lai, et al. StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistant. NeurIPS 2025.

Questions for authors
  • It is not clear what is new compared with AdaReTaKe combined with HERMES style retention and ReKV-style retrieval? Which component would be the main contribution, and how much does it add on its own?

  • How were B, K, W and M_KV chosen for each backbone? Why does the LLaVA result in Table 1 use K=64, W=6? Results with one shared config, or with settings tuned on a held-out split, would help a lot.

  • Why does accuracy drop so much at W=64? What does “author-reported” mean for the M_KV=2048 row?

  • It will be good to see a recent-window-only baseline, and a comparison of question-only retrieval against ReKV.

  • HERMES reports much lower TTFT in its own paper. It will be good to compare under matched conditions, i.e. no pending frames and the same KV budget.

  • It will be good to see mean and std over a few runs, and results on hour-long streams. The duration-independence claim is only really tested up to 1,024 frames.

  • The MVBench number (56.95) is the same as HERMES with 6K. It is not clear how this score was obtained.

Reviewer_KGBnOpenReview ↗
Rating
3
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The authors review existing streaming video understanding methods based on visual-token compression and KV-cache management, and investigate how to preserve historical evidence under limited memory when future questions are unknown. This is a meaningful research direction. The authors identify the central challenge as "how to allocate limited capacity across arriving frames and accumulating evidence to preserve a history that supports diverse future questions." To address this, Section 4 introduces StreamingRETAKE, a training-free framework combining Adaptive Visual-Token Allocation, Joint KV Retention, and Question-Conditioned KV Retrieval. But the primary novelty appears to reside in the delayed episode-level selection of KV evidence, rather than in a fundamentally new memory representation, optimization principle, or mechanism for preserving information under unknown future questions.

Critical strengths

StreamingRETAKE explicitly considers token admission, temporary memory residency, long-term retention, and question-time context construction. The framework provides a coherent engineering design for coordinating these operations without modifying the pretrained backbone. The experiments cover four backbone configurations and comparisons with several existing methods. The paper also evaluates frame-processing deadlines, question-response latency, GPU memory usage, component ablations, and parameter sensitivity. These results provide initial evidence of practical benefits on two streaming video understanding benchmarks, StreamingBench and OVO-Bench.

Critical weaknesses

The proposed Adaptive Visual-Token Allocation is motivated by the empirical finding that "the 20% of frames that differ most from their predecessors account for about 65% of the total change." However, this finding is supported by only 32 videos (16 each from the StreamingBench benchmark and the Video-MME benchmark), each truncated to 90 seconds. Since visual change directly guides token-budget allocation, such limited empirical evidence raises concerns not only about the generality of this finding but also about the robustness of the proposed method across different video distributions. Moreover, training-free KV compression and short-/long-term memory management have been extensively explored in prior work. Although StreamingRETAKE integrates these techniques and improves accuracy on StreamingBench and OVO-Bench, its methodological novelty beyond delayed episode-level KV selection remains unclear. Its ability to preserve useful historical evidence in memory over longer streams is also insufficiently demonstrated, a concern further reinforced by the inconsistent performance on memory-intensive OVO-Bench's Backward Tracing tasks in Table S1.

Questions for authors

Q1. Could the authors further clarify the key methodological differences between StreamingRETAKE and existing approaches, particularly RETAKE and AdaRETAKE, which address visual redundancy and KV-cache compression; InfiniPot-V (NeurIPS 2025), which provides training-free, query-agnostic KV compression under bounded memory; and Thinking-QwenVL (ICLR 2026), which employs hierarchical visual-token aggregation for streaming memory management within the LLM? What constitutes the substantive technical advance beyond these existing mechanisms?

Q2. How does the proposed method reliably estimate semantic relevance between textual questions and historical visual evidence without additional training? Cross-modal similarity estimation is challenging even with explicit training. Can the authors further clarify and demonstrate how much semantic discriminability the training-free Query–KV matching actually provides, particularly whether it can distinguish relevant visual evidence for different questions?

Q3. The authors should extend the key component ablations to memory-intensive tasks, particularly OVO-Bench Backward Tracing (EPM and ASI), and evaluate performance over longer evidence-to-query intervals. This would help establish whether the proposed method genuinely improves persistent historical memory rather than primarily benefiting real-time visual perception.

Reviewer_obmUOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes StreamingReTAKE, a training-free framework for streaming video understanding that performs causal evidence budgeting through three coordinated components: (1) adaptive visual-token allocation, which uses a budget bank to transfer token capacity from redundant to informative frames; (2) joint KV retention, which delays long-term KV selection until episode closure so earlier and later evidence compete for a shared retention quota; and (3) question-conditioned KV retrieval, which combines persistent importance with question–key relevance to construct a compact answer context without modifying stored history. Extensive experiments demonstrate that StreamingRETAKE achieves competitive performance across multiple models while maintaining bounded memory footprints and sustaining real-time processing speeds.

Critical strengths
  1. The paper addresses a practical challenge in streaming video understanding, where future questions are unknown and early retention decisions may discard useful evidence. The proposed adaptive token allocation and delayed KV retention are well motivated by this setting.
  2. The proposed framework is not only training-free and integrable into various MLLMs, but it is also backed by theoretical bounds on visual-history KV storage. In addition, the authors validate these guarantees with a comprehensive efficiency analysis (TTFT, backlog, physical GPU memory peak) that goes beyond standard accuracy metrics.
Critical weaknesses
  1. The method uses visual novelty and importance scores to select tokens, but these may not always capture what future questions need. For example, small visual changes could contain important details that get discarded.
  2. Joint retention only delays selection within a fixed window, while older evidence can still be evicted as the video grows. It would be useful to see how the method performs when questions require information from much earlier in the video.
Questions for authors

See weaknesses and:

  1. Joint KV retention only delays the selection decision within a fixed window of W frames. Evidence discarded at episode closure cannot be recovered later. It would be helpful to see how accuracy changes when questions refer to events much earlier in the video.
  2. As noted in [1], retaining only recent frames can already achieve high accuracy on StreamingBench and OVO-Bench. Since this method also prioritizes recent frames, it is unclear how much of the improvement comes from the proposed long-term memory mechanism. A recent-only baseline would help clarify this.
  3. Why were the Qwen-VL series models not included in the offline video benchmarks?

[1] A Simple Baseline for Streaming Video Understanding

1 comment

Thank you for the author's explanation, which has resolved some of my doubts. I will raise my score, but I may not be confident.

Comments on the paper

1 comment
Area_Chair_HA9uOpenReview ↗

rebuttal ddl

Dear reviewers, the ICLR deadline is only 4 days away. The authors have already rebuttaled the issues raised. If you still have questions, please participate in the rebuttal.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.