- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper argues that streaming video LLM look good on OVO-Bench/StreamingBench only because the evaluator decides when to ask questions. When models have to decide themselves when to answer, then their performance drop. The authors propose StreamAct-Bench and formalize a per-chunk policy with state, readiness, action and an answer. The authors also propose StreamScore (mean of State, Ready, Action, VFR). They evaluate around 11 models under a strict-causal replay interface and fine-tune Qwen3-Omni on 60K trajectories. Oracle experiments suggest that timing is more critical than perception in terms of performance.
Critical strengths
-
This topic is very timely and also interesting for several real-world applications. Most of video based applications will face streaming setup and whether models are appropriate for streaming scenario or not is important to study.
-
Paired evaluator vs self-timed analysis on same items is useful as it isolates timing form perception. Per-chunk decision labels in addition to performance on final answer are also useful for analyzing models behavior.
Critical weaknesses
-
The contributions of this work are very weak. The first contribution has been observed in several prior works such as StreamReady etc., where response timing is explored. The fact that explicit supervision is not super helpful, does not add any value to the list of contributions. Also, why just one backbone was tested?
-
The evaluation setup for replay is not well generalized. Ideally, memory based streaming models such as MMDuet, HiearQ, VideoLLM-online, ReKV, etc. are ideal test bed for this setup. Whether models can answer late or not will depend a lot on how good the memory is.
-
The figures are so hard to read and understand. The presentation needs a lot of work in this submission. I was not able to read most of the important figures.
Questions for authors
-
The authors should evaluate the streaming models in their own native mode, memory/cache/etc., instead of using a specialized replay setup.
-
Can you share details on self-timed prompt? Was it a generalized prompt for all?
-
In the annotation, was there any analysis on overlap between llm vs human and was any inter-annotator agreement computed?
-
Can we get evaluations on some closed-set models too?
-
Why is Full SFT VFR 11.11 in Table 3 but 8.84 in Table 4? Any reason why these are not on same shared trajectories?
-
The writing of this manuscript is very convoluted, very hard to understand. This is a very interesting topic, but the way it has been written, its hard to understand what the authors want to convey.
Some missing references:
-
Chen et al., VideoLLM-online: Online Video Large Language Model for Streaming Video, CVPR 2024.
-
Wang et al., VideoLLM Knows When to Speak: Enhancing Time-Sensitive Video Comprehension with Video-Text Duet Interaction Format (MMDuet), Findings of EMNLP 2025.
-
Wang et al., Enhancing Proactive Interaction of Video MLLMs with Multi-Turn Reinforcement Learning, ICLR 2026.
-
Qian et al., Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reaction, CVPR 2025.
-
S. Azad, V. Vineet, Y. S. Rawat. HierarQ: Task-Aware Hierarchical Q-Former for Enhanced Video Understanding. CVPR 2025.
-
Wang et al., StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistant, NeurIPS 2025.
-
Li et al., LION-FS: Fast & Slow Video-Language Thinker as Online Video Assistant, CVPR 2025.
8 comments
Weakness 2. Replay generality and native memory in MMDuet, HierarQ, VideoLLM-online, and ReKV
Thank you for this important suggestion. The StreamAct-Bench main leaderboard already includes recent streaming models such as ThinkStream-3B and MOSS-VL-Realtime. In response to your suggestion about native memory mechanisms, we added MMDuet and VideoLLM-online, two earlier streaming models that represent different capabilities. Together, the four models cover streaming reasoning and online state, real-time streaming control, proactive interaction and action selection, and online causal memory in the native-mode audit. Each model is evaluated on the same dataset, at the same decision nodes, and with the same scorer under both the unified explicit-state interface and the native interface that retains its KV cache, recurrent state, or memory bank. A lightweight adapter only serializes the structured fields. Neither mode exposes reference states, action labels, response windows, or future observations. With native memory, StreamScore increases modestly (+1.27 to +4.57), VFR remains generally low, and Early decreases slightly. The ranking is identical to that under the explicit interface: ThinkStream-3B > MOSS-VL-Realtime > VideoLLM-online > MMDuet, with 0 changes in the top 3. The native interface yields limited gains, leaves the rankings unchanged, and leaves timing calibration unresolved.
We agree that memory is essential to answering later in a stream. Models also need to learn when to respond proactively: they should trigger a response when evidence is sufficient and wait or remain silent when it is insufficient. A model can retain more history and still be uncertain about when to respond. StreamAct evaluates state and memory separately from readiness and action. In our results, the transition from evidence to action poses a greater bottleneck than memory capacity. The main leaderboard uses a unified explicit-state interface to diagnose proactive response under a common memory interface. The native-mode audit further shows that the rankings remain unchanged and timing calibration remains a problem when models retain native memory. These results support the central conclusion that answerability alone does not determine online behavior. Streaming intelligence requires state, readiness, action, and answers to work together across the trajectory. We will clarify the relationship between memory and proactive response in the revision and scope the leaderboard findings to controlled diagnosis under the unified explicit-state interface.
| Model | Interface | State↑ | Ready↑ | Action↑ | VFR↑ | Early↓ | Miss↓ | StreamScore↑ | ΔStream. |
|---|---|---|---|---|---|---|---|---|---|
| ThinkStream-3B | Explicit | 28.36 | 52.28 | 22.69 | 2.19 | 35.23 | 60.50 | 26.38 | - |
| Native | 29.74 | 54.86 | 33.92 | 5.28 | 34.91 | 29.14 | 30.95 | +4.57 | |
| MOSS-VL-Realtime | Explicit | 21.64 | 48.06 | 27.85 | 2.21 | 33.84 | 58.83 | 24.94 | - |
| Native | 23.41 | 51.37 | 36.12 | 5.82 | 33.41 | 57.36 | 29.18 | +4.24 | |
| VideoLLM-online | Explicit | 27.00 | 47.00 | 21.50 | 1.30 | 30.57 | 65.03 | 24.20 | - |
| Native | 27.12 | 47.91 | 27.76 | 2.45 | 30.21 | 64.38 | 26.31 | +2.11 | |
| MMDuet | Explicit | 24.37 | 46.85 | 20.94 | 2.63 | 38.21 | 69.74 | 23.70 | - |
| Native | 25.06 | 47.73 | 24.12 | 2.97 | 37.84 | 68.92 | 24.97 | +1.27 |
Weakness 3. Figure readability and clarity of writing
We take your criticism in Weakness 3 seriously. Several figures and tables are too dense, and some labels and legends need to be clearer. Parts of the text are also compressed, making it harder for readers to follow the method and interpret the conclusions. We appreciate you pointing this out. In the revision, we will systematically review the figures, tables, and organization of the text to present the key information more clearly and make it easier to understand.
Q1. Can streaming models be evaluated in their native modes?
Thank you for the suggestion. As explained in our response to Weakness 2, the revision adds a native-mode audit for ThinkStream-3B, MOSS-VL-Realtime, VideoLLM-online, and MMDuet. We run each model under the unified explicit-state interface and under its native interface, retaining the KV cache, recurrent state, or memory bank as appropriate. A lightweight adapter only serializes the structured fields, and future observations are withheld. Under the native interface, StreamScore increases modestly (+1.27 to +4.57), VFR remains generally low, Early decreases slightly, and the ranking is identical to that under the explicit interface. Native memory yields limited gains and leaves timing calibration unresolved. We therefore scope the leaderboard conclusions to controlled diagnosis of online behavior under the unified explicit-state interface. The revision will report both protocols' results and paired confidence intervals.
Q2. What is the self-timed prompt, and do all models use the same prompt?
Thank you for asking about the prompt. The paper uses two related settings with different output templates. Both follow the same strict causality constraints and use the same style of general instructions.
First, the paired evidence-boundary diagnostic in Section 2.2 uses self-timed replay. The model receives only the current video chunk and the question, chooses when to make its first response, and outputs its answer. This setting does not require structured state, readiness, or action fields. The full prompt follows.
You are observing a video stream incrementally. At each step, you receive only the current video chunk and the question. You do not have access to future frames, reference answers, or any annotation about when to respond.
Question: {question}
Options: {options}
Decide on your own whether the evidence observed so far is sufficient to answer the question.
- If the evidence is sufficient, output the answer immediately using exactly this format:
ANSWER: {option_label}
- If the evidence is not yet sufficient, output exactly:
WAIT
Do not explain your reasoning. Do not output anything else. Once you emit ANSWER, your response is final and will be scored as your first response.
Second, StreamAct-Bench's strict causal replay at each decision node requires the model to output state_phase, state, readiness, action, and answer. The state field has a hard budget of 128 tokens. The full prompt follows.
You are observing a video stream incrementally under strict causality. At each replay node, you receive only the current media chunk, the task objective, the question and options, and your own previously emitted state. You do not have access to future observations, reference answers, action labels, response windows, full media prefixes, or native hidden caches.
Objective: {objective}
Question: {question}
Options: {options}
Previous carried state: {previous_state}
Observe only the current media chunk. Update the carried state if needed, judge whether the evidence is sufficient, choose an action, and answer only when warranted. Return one compact JSON object with exactly these fields:
{
"state_phase": "before" | "changing" | "after" | "na",
"state": "concise task-relevant summary, at most 128 tokens",
"readiness": <number between 0 and 1>,
"action": "wait" | "silence" | "trigger" | "answer",
"answer": "<option label A-D>" | null
}
Rules:
- Use "wait" when evidence is not yet sufficient but may become sufficient later.
- Use "silence" when no response is warranted under the task condition.
- Use "trigger" when the instructed condition occurs and a proactive response is required.
- Use "answer" only when the evidence supports a specific option; otherwise set answer to null.
- Do not use future observations, reference answers, action labels, or response windows.
- Return only the JSON object, with no additional text.
All models within each setting use the same general prompt. Model-specific wrappers only serialize the media input and output schema. They leave the instructions identical across models.
We thank the reviewer for taking the time to read the paper and provide detailed feedback. We take seriously your concerns about the scope of the contribution, the generality of the replay setup, presentation quality, and reproducibility. We will address each in the revision and respond to the main weaknesses and questions below.
Weakness 1. Contribution beyond prior work, benefits of explicit supervision, and backbone coverage
We agree that StreamReady and related work have identified the problem of when to respond. StreamAct addresses a broader set of streaming behaviors. Our first contribution is to identify what accounts for the gap in current evaluations. Most video benchmarks measure answer correctness at a time selected by the evaluator, leaving the joint behavior of state maintenance, evidence readiness, action selection, and conditional emission across the full timeline unmeasured. Our second contribution is a trajectory-level evaluation that formalizes how state, readiness, action, and answers evolve chunk by chunk. We construct 5,436 replayable trajectories and 91,839 decision nodes and use strict causal replay to locate the largest gap in the transition from evidence to action. StreamReady focuses on when evidence supports an answer. StreamAct additionally audits state revision and the decisions to wait, remain silent, trigger, or answer. Our contribution is to identify and attribute this measurement gap and provide a complete, auditable trajectory evaluation that also supports training.
We would like to clarify the role of explicit supervision. As the main text and conclusion state, StreamAct-SFT is the supervised intervention in a framework that connects diagnosis, evaluation, and training. It tests whether the identified state, readiness, and action behaviors can be learned. On Qwen3-Omni, Full SFT changes Action by +17.35, Miss by −24.36, and StreamScore by +5.74, showing that the process signals are learnable and that missed responses decrease substantially. VFR improves by only +4.43, and Early increases by +13.97, so timing calibration remains unresolved. This trade-off between conservative waiting and proactive responding is a finding of the paper. Explicit process supervision improves action selection and reduces missed responses, while aligning readiness with response emission still requires work on the architecture, training objectives, or response controller. The diagnostic framework makes these remaining gaps measurable and auditable. We have also added experiments on Qwen3-VL-30B-A3B, InternVL3.5-30B-A3B, and InternVL3.5-20B-A4B. Full SFT improves Action, StreamScore, and VFR and reduces Miss on all three backbones. The direction of the change in Early depends on the backbone's initial action policy. The results show that the supervision signal transfers beyond the Qwen family. They also support reporting timing calibration by backbone and treating timing control as a remaining challenge for architecture and training design.