acceptodds
Under review as a conference paper at ICLR 2027

DualStreamBench: A Benchmark for Dual-System Streaming Video Understanding

Abstract

A streaming video assistant must decide when a question warrants more computation. We propose DUALSTREAMBENCH, a benchmark that evaluates this decision through paired fast and slow answers to the same timestamped query. It covers 5,619 questions over 1,392 videos, 2,190 new human-authored open-answer questions, and audited evidence-demand labels across diverse streaming tasks. Our central finding is that paired diagnosis changes resource and routing decisions that aggregate accuracy alone cannot support. Across closed- and open-source systems, including open models up to 397B parameters, slow paths overturn correct fast answers on 3.7–14.5% of questions, including 6.7–13.1% in the six full-core configurations where they raise average accuracy. Training on paired outcomes improves allocation: on a 35B model, a rescue router beats random selection given twice as many slow calls and comes within 0.02 points of always escalating with 25% fewer, whereas a frozen visual gate stays near random. On the same checkpoint, a longer visual window adds 6.57 points while 40.8× more thinking tokens add none, and complete-stream replay attributes 91.6% of routed service cost to memory maintenance. We believe DUALSTREAMBENCH makes beneficial escalation, transferable allocation, and state maintenance measurable targets for streaming video systems.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
video understanding, streaming, MLLM

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_yk4mOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper builds DualStreamBench: 5619 timestamped questions over 1392 video, reannotated from StreamingBench, OVBench, CG-Bench and OVO-Bench, along with 2190 new open-ended questions scored by LLMs. Each question carries a human label for whether it needs fast perception or history/reasoning. The core protocol runs a fast path (5 sec window, direct answer) and a slow path (50 sec window, memory retrieval, thinking) on every query. The accuracy change is split into rescue and harm, and routing policies are evaluated under a compute budget.

Critical strengths
  • Paired per-question rescue/harm accounting with human labels: This is useful because just looking at average accuracy can not show if “more context/thinking” breaks questions for cases where the fast path already got right. The D1/D2 labels are a useful resource for studying adaptive compute in streaming videos.

  • End-to-end cost accounting: The full-stream replay showing that memory maintenance is around 90% of routed cost is an interesting result in the paper. It is a good indication that query-time routing is not important for savings in memory-based streaming systems.

Critical weaknesses
  • The routing problem looks artificial: A real streaming assistant does not have to choose between fast and slow per query. It can answer immediately from the recent window and let the slow branch (or an agent) run asynchronously and revise the answer when it finishes. Two-pass streaming ASR has explored this (Dispider), and VST/ThinkStream has thinking during playback. If compute cost is the concern, then there are several other important pieces in streaming video understanding, which should be optimized.

  • Better baselines missing: The paper compares only always-fast, always-slow, random, learned routers and an oracle. It has no fast-then-revise baseline, no always-parallel slow branch, and no agentic tool-calling baseline, and the cost model counts invocations and latency but not user-facing time-to-first-answer or revision cost.

  • It seems that query service is a small share of total cost and that accuracy is similar across policies. This suggests that “always run slow in the background” could be a better choice. The paper never tests this and therefore it is not clear if budgeted routing is required.

  • Limited novelty: re-packaged data: The benchmark is merge and re-annotation of four public benchmarks, and the new open-ended questions are on existing videos. That can also cause contamination issues for recent models.

  • Not so surprising findings: The fast/slow framing for online video assistants already exists (LION-FS, StreamMind, Dispider), and also the recent-window baseline (SimpleStream).

Questions for authors
  • It would be good to see a policy that always gives the fast answer first and then a revised slow answer. It would help to score this on the final answer and on the time to the first answer. If revision matches always-slow accuracy at fast-path latency, it would be useful to explain what is left of the routing motivation.

  • Did the authors analyze the effects of the window, memory and thinking separately? It would help to run a 50 s window without thinking, a 5 s window with thinking, and memory alone, with rescue and harm reported for each. Also, please rerun thinking without the token cap, or to report results on non-truncated outputs only.

  • The authors should clarify whether response timing is scored for Proactive questions. If it is not, please explain how this mode differs from QA at a fixed timestamp.

  • The authors should also discuss how much of the source data the 2025-2026 models could have seen in training. Can we check whether the conclusions hold on the 2,190 new questions alone.

Reviewer_fMHeOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

Existing streaming video benchmarks focus on answer accuracy but overlook when additional computation is beneficial. This paper introduces DualStreamBench, which evaluates paired fast and slow inference through rescue, harm, and budgeted routing across diverse video understanding tasks. Experiments across multiple models show that slow inference can both improve and degrade accuracy, while learned routing provides modest gains and visual information offers little additional benefit over text-only routing.

Critical strengths
  1. The benchmark covers diverse streaming video understanding tasks, with 5,619 timestamped questions across five operating modes and 20 task types, supplemented by 2,190 human-authored open-answer questions.

  2. The paper evaluates a wide range of closed- and open-source models, with detailed analyses of fast/slow inference, routing strategies, resource allocation, and memory maintenance costs.

Critical weaknesses
  1. Limited novelty of the benchmark. The core dataset is assembled from four existing benchmarks, and the task taxonomy largely reorganizes established video understanding tasks. The main additions are demand annotations and paired fast/slow evaluation, rather than new tasks or evaluation scenarios.

  2. Lack of evaluation on native streaming video models. The experiments rely on general-purpose multimodal models with fixed video windows and external memory pipelines. No dedicated streaming video models are evaluated, despite streaming video understanding being the central focus of the paper.

  3. Gap between the dual-system motivation and experimental design. The main routing experiments distinguish fast and slow inference simply by using 5-second versus 50-second visual contexts, without changing the reasoning process. Moreover, D2 combines historical retrieval and complex reasoning, although these require different resources. This weakens the connection between the proposed dual-system formulation and its actual evaluation.

  4. Limited practical benefit of visual routing. The learned visual router improves accuracy by only 0.05 percentage points over the text-only router, with no statistically significant difference, despite requiring additional visual processing. Routing performance also varies considerably across datasets, limiting its practical applicability.

Questions for authors
  1. Could the authors analyze whether response length or style affects LLM judge decisions, particularly the reported rescue and harm rates, using the existing human-annotated samples?

  2. Could the authors clarify how outputs reaching the generation token limit affect the thinking accuracy results in Table 13, particularly by comparing completed and truncated responses?

Reviewer_ZKWbOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies when a streaming video model should spend more computation on a question. It compares fast and slow answers to identify when extra computation fixes mistakes or makes correct answers wrong. Its main contributions are a benchmark, an evaluation protocol, and routing baselines that decide which questions need extra resources. Experiments across multiple models show that slower processing does not always help. Learning from paired answers improves resource allocation, while longer video context can help more than extra thinking. The tested system also spends most of its time maintaining memory.

Critical strengths
  1. The research motivation of the paper is meaningful. The paired evaluation measures both corrections and errors introduced by extra computation. This reveals information that average accuracy alone cannot provide.
  2. The benchmark covers diverse streaming tasks and models of different sizes. It also considers memory maintenance costs, making the evaluation more relevant to practical streaming systems.
Critical weaknesses
  1. The core benchmark is primarily constructed by integrating four existing datasets, with relatively limited new annotations. Moreover, the median of only three questions per video raises concerns about whether the benchmark adequately evaluates sustained long-term streaming video understanding.
  2. When only text is used, the accuracy rate of multiple-choice questions reaches 40%-42%, indicating that the model may utilize linguistic prior knowledge and biases in answer options, rather than relying on video comprehension. The quality of the data remains to be verified.
Questions for authors

See weaknesses and:

  1. The full-stream replay uses a fixed 60-second interval for memory updates, while memory maintenance accounts for 91.59% of the routed service time. It would be helpful to see how different memory update frequencies affect accuracy and computational cost.
  2. The routing strategies show limited generalization across datasets, with some results falling below random routing. What factors contribute to this performance degradation, and how does it affect the reliability of DualStreamBench for evaluating adaptive computation strategies?
  3. In Table 8, memory appears to provide less benefit on D2 questions than D1 questions for some larger models, such as Qwen3.5-122B and 397B. It would be helpful to provide further analysis or discussion of this somewhat counterintuitive result.
1 comment

Thank you to the authors for their response and rebuttal. Although I still believe there are certain limitations regarding the generalization of the dataset, I am inclined to maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.