- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper builds DualStreamBench: 5619 timestamped questions over 1392 video, reannotated from StreamingBench, OVBench, CG-Bench and OVO-Bench, along with 2190 new open-ended questions scored by LLMs. Each question carries a human label for whether it needs fast perception or history/reasoning. The core protocol runs a fast path (5 sec window, direct answer) and a slow path (50 sec window, memory retrieval, thinking) on every query. The accuracy change is split into rescue and harm, and routing policies are evaluated under a compute budget.
Critical strengths
-
Paired per-question rescue/harm accounting with human labels: This is useful because just looking at average accuracy can not show if “more context/thinking” breaks questions for cases where the fast path already got right. The D1/D2 labels are a useful resource for studying adaptive compute in streaming videos.
-
End-to-end cost accounting: The full-stream replay showing that memory maintenance is around 90% of routed cost is an interesting result in the paper. It is a good indication that query-time routing is not important for savings in memory-based streaming systems.
Critical weaknesses
-
The routing problem looks artificial: A real streaming assistant does not have to choose between fast and slow per query. It can answer immediately from the recent window and let the slow branch (or an agent) run asynchronously and revise the answer when it finishes. Two-pass streaming ASR has explored this (Dispider), and VST/ThinkStream has thinking during playback. If compute cost is the concern, then there are several other important pieces in streaming video understanding, which should be optimized.
-
Better baselines missing: The paper compares only always-fast, always-slow, random, learned routers and an oracle. It has no fast-then-revise baseline, no always-parallel slow branch, and no agentic tool-calling baseline, and the cost model counts invocations and latency but not user-facing time-to-first-answer or revision cost.
-
It seems that query service is a small share of total cost and that accuracy is similar across policies. This suggests that “always run slow in the background” could be a better choice. The paper never tests this and therefore it is not clear if budgeted routing is required.
-
Limited novelty: re-packaged data: The benchmark is merge and re-annotation of four public benchmarks, and the new open-ended questions are on existing videos. That can also cause contamination issues for recent models.
-
Not so surprising findings: The fast/slow framing for online video assistants already exists (LION-FS, StreamMind, Dispider), and also the recent-window baseline (SimpleStream).
Questions for authors
-
It would be good to see a policy that always gives the fast answer first and then a revised slow answer. It would help to score this on the final answer and on the time to the first answer. If revision matches always-slow accuracy at fast-path latency, it would be useful to explain what is left of the routing motivation.
-
Did the authors analyze the effects of the window, memory and thinking separately? It would help to run a 50 s window without thinking, a 5 s window with thinking, and memory alone, with rescue and harm reported for each. Also, please rerun thinking without the token cap, or to report results on non-truncated outputs only.
-
The authors should clarify whether response timing is scored for Proactive questions. If it is not, please explain how this mode differs from QA at a fixed timestamp.
-
The authors should also discuss how much of the source data the 2025-2026 models could have seen in training. Can we check whether the conclusions hold on the 2,190 new questions alone.
Thank you to the authors for their response and rebuttal. Although I still believe there are certain limitations regarding the generalization of the dataset, I am inclined to maintain my original score.