- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces AskStream, the first multi-turn benchmark spanning video and user uncertainty for streaming video understanding, to test whether the model can proactively ask for unseen information when doing video comprehension. Experiments shows that current video multimodal foundation models struggles with this task with the highest F1 score of 52.18%.
Critical strengths
-
The task about asking questions during streaming video comprehension is novel, interesting, and critical for real world video comprehension.
-
The authors carried out solid experiments across both closed-source models, open-source models and streaming proactive models, showing that current video foundation model struggles with the proposed task.
-
The curation process of the benchmark is well-designed. Timestamp windows are labeled by hand with leakage check on the masked video. The author also manually validated the judge consistency between the human annotators.
Critical weaknesses
-
Form the way the benchmark is curated, for each sample, in order to successfully answer the question, the model must ask before answering. This would encourage the model to ask even it should not ask.
-
The analysis all relied on Gemini model.
-
There is no pure text baseline. It's possible that the model could answer the question without watching the video.
Questions for authors
-
The performance of a lot of streaming models are 0 across the benchmark. Is the model truly not able to complete the task, or it's a prompting or instruction following problem?
-
There a typos in model's respond in Figure3. Like "The branch is Cuisinart". Current LLM based foundation models is very unlikely to make this kind of typos. Could the authors clarify how these responses are generated?
5 comments
Response to Reviewer p5c4
Q1: Analysis of the Poor Performance of Streaming Models.
Thank you for the question. Existing streaming models are mainly trained with special tokens such as <wait> and <response> to represent silence and responses, respectively. For a fair comparison, we map <wait> to WAIT and <response> to ASK or ANSWER. We then use regular expressions to determine whether the generated response corresponds to ASK or ANSWER. The errors of Instruction following are rare (2.3% in both settings). The dominant failure mode is remaining silent throughout the entire turn (75.6%). This suggests that current models may not have fully learned the interaction policies needed for proactive streaming tasks. One possible explanation is the limited generalizability of existing training methods or datasets.
A simple improvement would be to introduce an <Ask> token into the model vocabulary and provide corresponding training examples. More broadly, future work could develop training strategies that help models generalize their decisions about when to act across a wider range of interactions, beyond ASK and ANSWER.
Q2: Clarification of the Typos in Figure 3.
Thank you for pointing this out. We apologize for the typo in Figure 3: “The branch is Cuisinart” should read “The brand is Cuisinart.” This typo was introduced by us during manual figure preparation. We have already corrected this error in Figure 3.
We thank the reviewer again for the thoughtful and constructive feedback, which has helped us improve the clarity and rigor of our work.
Response to Reviewer p5c4
Dear Reviewer p5c4,
Thanks for your time and effort in reviewing this paper. Your comments are valuable to improve the quality of this paper. As the author-reviewer discussion period will end in about 2 days, we gently request that you read the rebuttal and check whether your concerns are addressed. Thanks again for your time and help in improving the quality of this paper.
Regards from the authors.
Response to Reviewer p5c4
Thank you for your careful and insightful reviews. We appreciate your recognition of the novelty and real-world importance of proactive asking in streaming video understanding, the comprehensiveness of our experiments, and the rigor of our benchmark curation, as well as your constructive comments on the benchmark design, analysis, and baseline comparisons. We sincerely hope that our response addresses your concerns.
Thank you for raising the concern that our evaluation protocol may encourage models to ask unnecessary questions. Our protocol was motivated by a limitation of ProactiveBench. In that work, the prompts do not explicitly inform models that they can ask questions, resulting in near-zero asking rates across models, including GPT models. We believe this approach does not adequately measure models’ ability to ask questions proactively. Therefore, in our setting, the prompt explicitly states that information may or may not be missing and allows the model to choose whether to ask. This formulation is intended to provide a more objective assessment of proactive asking.
To investigate whether our protocol introduces a bias toward asking, we evaluated gpt-4o on the PO proactive subset of StreamingBench, using StreamHarness and our existing evaluation method, with only a minor modification allowing the model to answer directly without first issuing an ASK. This subset contains no missing information and therefore requires no asking. As shown below, the unnecessary ask rate was 0.00%, indicating that our protocol does not induce unnecessary asking. Meanwhile, accuracy increased from 56.86% to 70.59%, further supporting our protocol’s ability to more accurately assess offline models’ proactive understanding capabilities.
Thank you for highlighting this limitation. We extended the oracle ablation to GPT-5.6-Sol and Qwen-3.8 Flash under the same settings. As shown in the table below, WHEN_ASK and WHEN_WHAT_ASK improve Ask-F1 over NONE by 17.73 and 40.64 percentage points for GPT-5.6-Sol, and 24.43 and 46.29 points for Qwen-3.8 Flash, respectively. Both models show the same main trend as Gemini-3.8 Flash, supporting when and what to ask as major bottlenecks across the evaluated models.
Thank you for highlighting the missing text-only baseline. During benchmark construction, human annotators verified all eight subtasks under two conditions: without and with the requested evidence (text, images, or video). Even with access to the original video, the answer could not be uniquely determined without the additional evidence, whereas providing it resolved the uncertainty (Section A.7.2).
To directly assess textual shortcuts, we additionally evaluate Gemini-3.8-flash using only the question and candidate options, independently for the ASK and ANSWER stages. We report accuracy under three settings: original option order, shuffled order, and shuffled order with text rephrased by Qwen3.8-flash. As shown in the table below, both ASK and ANSWER accuracies remain close to the random-selection baseline across all three settings. These results suggest that textual cues alone provide limited predictive value.