acceptodds
Withdrawn from ICLR 2027

Query-adaptive Visual-semantic Evidence Allocation for Streaming Video Understanding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Streaming video understanding requires answering questions over an ever-growing visual history under limited memory and inference budgets. Existing methods typically manage this history through fixed-capacity memory compression or similarity-based frame retrieval, but largely rely on fixed compression or retrieval criteria without explicitly modeling the specific historical evidence requirements of each query. This overlooks a key property of streaming questions: their historical evidence requirements are inherently heterogeneous. To address this issue, we propose a training-free framework for query-adaptive visual-semantic evidence allocation that jointly determines whether to access history, which temporal forms of visual evidence to prioritize, and which long-range semantic evidence to retrieve. Given a query and recent observations, the framework jointly predicts whether historical access is needed and the query’s preferences over static states, dynamic context, and recency-sensitive evidence. Questions supported by recent obser- vations are answered directly through Fast–Slow routing. For queries requiring history, the same evidence preferences guide both fine-grained visual selection and event-level semantic ranking. The visual branch combines query-specific preferences for static, dynamic, and recency-sensitive evidence to select a compact and diverse set of historical frames. The semantic branch propagates the preference signals to events in a hierarchical caption memory, fuses the event scores with the same query preferences, and complements preference-guided ranking with VLM- based caption retrieval. The resulting visual and semantic evidence is integrated with recent observations for final answering. Our framework achieves 67.79% accuracy on OVO-Bench and 77.01% across the 14 visual-only tasks of Streaming- Bench, outperforming the compared methods on both benchmarks. These results demonstrate the value of coordinating history access and visual-semantic evidence selection according to query-specific requirements.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Streaming Video Understanding, Multimodal Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_BfjyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a training-free framework for query-adaptive visual-semantic evidence allocation. Given a query and recent observations, the framework jointly predicts whether historical data needs to be accessed, as well as the query’s preferences over static states, dynamic context, and recency-sensitive evidence. The experimental results on OVO-Bench and StreamingBench demonstrate the effectiveness of the proposed method.

Critical strengths
  1. The paper is well written and the figures are nicely drawn. It is easy to follow.
Critical weaknesses
  1. The paper is only tested on OVO-Bench and StreamingBench. Should test on newer streaming video benchmarks, such as OVO timing.
  2. Recency-sensitive evidence corresponds to the Forward tasks, but from A1 to A2, REC does not change at all.
Questions for authors
  1. Could you add the experimental results of your method on OVO timing?
  2. Could you explain why REC does not improve from A1 to A2?
Reviewer_zFyoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on the task of streaming video understanding, and proposes a training-free framework that adapts historical evidence retrieval to different query requirements. It first uses Fast–Slow routing to determine whether historical information is needed, and predicts preferences over static, dynamic, and recency-sensitive evidence to guide visual and semantic retrieval. Experiments on OVO-Bench and StreamingBench show strong performance.

Critical strengths
  • The proposed method achieves competitive results on both OVO-Bench and StreamingBench. The experiments also include multiple backbone architectures, including Qwen2.5-VL, Qwen3-VL, and LLaVA-OneVision-2, providing evidence of its transferability.
Critical weaknesses
  • Several key ideas, including query-based routing, temporal retrieval, and hierarchical event memory, have already been explored in recent works such as OASIS (CVPR 2026), StreamRAG (CVPR 2026), and DiG (CVPR 2026). The main difference seems to be the use of shared query preferences for visual and semantic retrieval, but I am not convinced that this is a sufficiently substantial contribution. The paper also lacks experimental comparisons with OASIS, StreamRAG and DiG in Tables 1 and 2, making it harder to assess its advantages over these closely related methods.
  • Many components of the framework rely on hand-crafted heuristics and manually designed prompts. While these choices seem reasonable and the overall results are promising, there is limited justification for why they are designed this way. More importantly, it is unclear whether the improvements actually come from modeling different evidence requirements, as claimed, or simply from providing more historical visual and semantic information. The current ablations do not fully disentangle these factors.
Questions for authors
  • What motivates the definitions of static, dynamic, and recency-sensitive evidence? Since these preferences are predicted through manually designed prompts, how sensitive is the final performance to different prompt formulations?
  • How accurate are the Fast–Slow routing decisions? What happens when a query is incorrectly routed to Fast, especially when historical evidence is actually needed?
  • Is it reasonable to assume that more recent frames are more important? Shouldn't this depend on the specific query?
  • Isn't the proposed dynamic relevance essentially just expanding the temporal neighborhood around relevant frames?
Reviewer_x21mOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a training-free streaming video understanding pipeline, which adaptively determines whether historical evidence is needed. It combines Fast–Slow routing, query-specific visual evidence preferences, and hierarchical semantic memory retrieval. Experiments on OVO-Bench and StreamingBench report improved performance over existing methods.

Critical strengths

The framework addresses the heterogeneous evidence requirements of streaming queries by explicitly coordinating recent observations, historical frames, and semantic memory.

Critical weaknesses

1.Limited methodological novelty. The framework primarily combines existing routing, similarity-based retrieval, temporal heuristics, and hierarchical caption memory. The contribution appears more system-level than algorithmically fundamental.

2.Unclear long-term efficiency and robustness. Although visual memory is bounded, textual memory can continue growing. The framework also depends heavily on accurate routing and generated captions, while its query-time latency remains substantial.

Questions for authors

How frequently does Fast–Slow routing incorrectly skip necessary historical evidence, and how does this affect downstream accuracy?

How do memory consumption, caption-generation costs, and end-to-end latency scale with substantially longer video streams?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.