acceptodds
Withdrawn from ICLR 2027

DeltaStreamer: Towards Human-Like Agentic Streaming Video Understanding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We propose DeltaStreamer, a novel agentic framework for streaming video understanding inspired by how humans observe, remember, and investigate. Streaming video understanding requires a system to retain evidence before knowing which details future queries will need. Existing approaches may obscure event transitions and omit relevant details when compressing video history, and cannot recover missing visual evidence. DeltaStreamer addresses these limitations by combining change-aware memory with query-driven planning and targeted evidence gathering with replay. Before a query arrives, it continuously observes the stream and explicitly records visual changes in structured memory, preserving how events change rather than only what is present. When a query arrives, a query-driven planner identifies the required evidence and guides memory retrieval and targeted replay. By revisiting historical frames, the agent can recover visual details omitted during observation and ground its answer in evidence beyond the retained descriptions. This connects continuous observation with active investigation tailored to each question. Experiments on OVO-Bench and StreamingBench demonstrate improvements over representative streaming video understanding methods, with overall scores of 66.07% and 59.53%, respectively.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
streaming video understanding, video agent

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_oJNGOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces DeltaStreamer, an agent-based framework for streaming video understanding. Specifically, before a question arrives, a vision-language model compares adjacent frames and writes structured observation reports, explicitly recording frame-to-frame deltas while filtering out redundant descriptions. These observations feed a traceable three-layer memory (working, episodic, and long-term pins) built as a DAG, with entity indexes and source pointers linking back to original frames. After a question arrives, a planner parses what the question really needs and creates an evidence contract, guiding memory retrieval. The agent can then replay historical frames to verify pixel-level details and uses a conflict verifier plus tools like SAM 2 and Whisper to correct errors. Experiments are conducted using Qwen3.6-27B as the backbone on OVO-Bench and StreamingBench datasets.

Critical strengths
  1. The article has a clear motivation, and both the writing and framework are well organized.
  2. The ablation studies demonstrate the effectiveness of each component.
Critical weaknesses
  1. Existing streaming video agents already include components such as memory, plan, and verify. This paper mainly adds direct inter-frame change recording, which makes the claimed innovation appear rather limited.
  2. The superiority of the proposed method is not demonstrated. The authors should explain why some results are missing for the limited open-source online MLLMs. Moreover, many of these models achieve higher results than the proposed method, yet these are not highlighted in bold in the tables. In addition, compared with prior agent work, the backbone models used are different, so comparisons should at least be conducted under the same backbone model.
  3. The compared baselines are outdated and insufficient. The proprietary MLLMs compared are all roughly two years old, which is too dated. The open-source offline MLLMs compared are either relatively small (7B–8B) or relatively old (only one Qwen2-VL-72B), and even the backbone model used in this paper, Qwen3.6-27B, is not compared. There are also many inconsistencies in method names between Table 1 and Table 2 (e.g., "human agents" in Table 1 vs. "human" in Table 2; gemini-3-flash appears in Table 1 but not in Table 2).
  4. Latency is not evaluated. Latency is critical for streaming video reasoning, yet the paper does not compare it.
Questions for authors

Please refer to the weaknesses.

Reviewer_UmQZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DeltaStreamer is an agentic framework for streaming video understanding that combines explicit inter-frame change descriptions, traceable hierarchical memory, and query-driven planning with targeted historical-frame replay. It records observations before questions arrive and subsequently retrieves or revisits relevant evidence to answer them. The method achieves overall scores of 66.07% on OVO-Bench and 59.53% on StreamingBench.

Critical strengths
  1. Well-motivated design. Explicit change descriptions preserve event transitions, while historical-frame replay recovers visual details omitted from textual memory. These mechanisms address complementary challenges in streaming video understanding.

  2. Useful controlled ablations. With identical two-frame inputs, explicit change recording improves OVO-Bench performance from 61.12 to 66.07 (Table 3), supporting its value beyond additional visual input.

  3. Detailed methodological documentation. The appendix describes memory construction, retrieval rules, tool interfaces, and major prompts. Quantitative ablations and qualitative examples help explain how the components contribute to answering.

Critical weaknesses
  1. It is not clear what the method adds beyond the closest prior work. The paper describes EventMemAgent as already using event memory, visual anchors, inter-event change logs, and visual tools, while StreamAgent already uses planning. DeltaStreamer compares adjacent frames directly and retains its plan across reasoning rounds, but the paper does not show how much these differences matter relative to those existing designs. Table 3 shows that explicit deltas help this system; it does not establish an advantage over prior change-aware memory methods.

  2. The main comparisons do not separate the framework’s gains from the backbone’s capabilities. DeltaStreamer uses Qwen3.6-27B, but there is no comparison with a close baseline running on the same backbone under comparable inference and tool budgets. It is also unclear which baseline scores were reproduced and which were taken from prior papers. This makes it difficult to tell how much of the improvement comes from the proposed design.

  3. The paper does not show whether the system can keep up with a live video stream. Each sampled frame requires a detailed observation, and answering can involve additional planning, retrieval, and replay calls. Yet response timing is measured in video timestamps rather than actual elapsed time. Figure 5 also measures only the text passed to the model, not all stored records and historical frames. Actual processing speed, response latency, and storage growth are needed to assess the practical cost of the accuracy gains.

Questions for authors
  1. What specific capability or mechanism distinguishes DeltaStreamer from the closest systems, particularly EventMemAgent and StreamAgent? Can a direct comparison isolate the contribution of explicit adjacent-frame changes and persistent planning beyond their existing memory and tool-use mechanisms?

  2. Can the authors provide matched-backbone comparisons under comparable inference budgets and information access, and clarify which baseline results were reproduced versus taken from prior work?

  3. What are the end-to-end processing throughput, query-response latency, and total storage growth on the reported hardware? How does the system handle incoming frames when inference falls behind the stream?

Reviewer_hC7vOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DeltaStreamer, an agentic framework for streaming video understanding. It targets two weaknesses of existing approaches: compressed memory records what is present but not what changes, and details discarded during observation cannot be recovered when a later query needs them. Before a query arrives, a VLM compares each pair of adjacent frames and produces a structured scene description plus an explicit textual “delta” of visual changes. These observations are stored in a three-layer hierarchical memory with cross-layer pointers and an entity index. When a query arrives, a planner generates a structured plan that specifies the target, temporal anchor, evidence type, and where the evidence likely lies; the plan persists across later rounds to guide retrieval, continued observation, and tool use. When textual memory lacks a needed detail, or a rule-based verifier flags conflicts between memory layers, the agent replays retained historical frames to recover pixel-level evidence.

Critical strengths
  1. The method is interesting, and the change-recording design is simple yet effective. By explicitly recording inter-frame changes, it captures finer details than caption-based summarization approaches.

  2. The problem is clearly formulated and the motivation is well articulated. The design avoids having important changes buried under the high redundancy between consecutive frames.

  3. The entire system is training-free, built purely from prompts, rules, and tools.

  4. The appendix is thorough and detailed, which supports good reproducibility.

Critical weaknesses
  1. DeltaStreamer is built on Qwen3.6-27B, whereas most baselines rely on smaller and older backbones. The paper lacks the most basic comparison: the performance of Qwen3.6-27B itself without the proposed framework. Without this baseline, it is unclear how much of the improvement comes from the framework rather than the stronger backbone.
  2. The method invokes the 27B model on every sampled frame, in addition to the planning step and multiple tool calls. However, no efficiency analysis is provided. Could the authors report end-to-end latency statistics as well as GPU memory consumption?
  3. The method lacks a proactive response mechanism.
  4. The method relies on many heuristic hyperparameters, yet no sensitivity analysis is provided.
Questions for authors

Replay is called only 0.17 times per question, yet its removal costs 2.79 points. Could you explain this?

Reviewer_GJo3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper attempts to address video understanding through the concept of VLMs and agents. Nevertheless, it seems to extend traditional vision tasks, addressing issues that are more matters of imagination than real-world challenges, without rigorous evaluation on established benchmark datasets.

Critical strengths

Explicit Change Awareness and Lightweight Memory

By computing and recording inter-frame visual deltas rather than mechanically piling up images and constructing a three-layer directed acyclic graph (DAG) memory architecture, it filters redundant information while accurately capturing key event evolutions.

Look-Back Verification and Anti-Hallucination Mechanism

When the retrieved textual memory lacks critical visual details, the system supports actively replaying historical raw frames and extracting pixel-level evidence for secondary cross-checking, effectively reducing the hallucination rate of large models.

Critical weaknesses

Lack of End-to-End Training and Model Weight Optimization

The framework does not re-finetune or re-train the underlying vision-language models (VLMs), relying entirely on the native capabilities of off-the-shelf base models combined with external prompts and tool scheduling, constraining its upper bound to the performance of the underlying APIs.

Lake of SoTA LLM Comparisons

This work does not include comparisons with the latest LLMs. Given the current requirements for computer vision tasks, it is recommended to incorporate the latest LLMs for benchmarking.

Questions for authors

None

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.