acceptodds
Under review as a conference paper at ICLR 2027

AskStream: When and What to Ask Proactively under Uncertainty in Streaming Video Understanding

Abstract

Recent advances in multimodal large language models have increased interest in interaction paradigms for video understanding. The evolution from offline to streaming and proactive interaction has gradually shifted models from passive responders toward more proactive agents that can decide when to answer. However, existing proactive paradigms typically assume that the information required to answer a query is complete, leaving models to only wait or answer. In the real world, however, information is often incomplete due to uncertainty from either the video or the user. In proactive paradigms, such uncertainty can cause the model to wait forever or generate highly hallucinatory responses. We therefore argue that proactive interaction should be redefined with a third action: ask, allowing models to actively ask for missing information. To study when and what models should ask before answering, we introduce AskStream, the first multi-turn benchmark spanning video and user uncertainty for streaming video understanding, four task categories, and eight subtasks. It contains 438 videos with 2,714 manually annotated frame-level time windows across three types: wait, ask, and answer. Across both multiple-choice and open-ended settings, even the strongest model attains only 52.18% on our proposed Ask-F1 metric, highlighting substantial limitations in proactive information seeking under uncertainty.

Primary area:
datasets and benchmarks
Keywords:
Streaming Video Understanding, Proactive Information Seeking, Multimodal Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_p5c4OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AskStream, the first multi-turn benchmark spanning video and user uncertainty for streaming video understanding, to test whether the model can proactively ask for unseen information when doing video comprehension. Experiments shows that current video multimodal foundation models struggles with this task with the highest F1 score of 52.18%.

Critical strengths
  1. The task about asking questions during streaming video comprehension is novel, interesting, and critical for real world video comprehension.

  2. The authors carried out solid experiments across both closed-source models, open-source models and streaming proactive models, showing that current video foundation model struggles with the proposed task.

  3. The curation process of the benchmark is well-designed. Timestamp windows are labeled by hand with leakage check on the masked video. The author also manually validated the judge consistency between the human annotators.

Critical weaknesses
  1. Form the way the benchmark is curated, for each sample, in order to successfully answer the question, the model must ask before answering. This would encourage the model to ask even it should not ask.

  2. The analysis all relied on Gemini model.

  3. There is no pure text baseline. It's possible that the model could answer the question without watching the video.

Questions for authors
  1. The performance of a lot of streaming models are 0 across the benchmark. Is the model truly not able to complete the task, or it's a prompting or instruction following problem?

  2. There a typos in model's respond in Figure3. Like "The branch is Cuisinart". Current LLM based foundation models is very unlikely to make this kind of typos. Could the authors clarify how these responses are generated?

5 comments
AuthorsOpenReview ↗

Response to Reviewer p5c4

Thank you for your careful and insightful reviews. We appreciate your recognition of the novelty and real-world importance of proactive asking in streaming video understanding, the comprehensiveness of our experiments, and the rigor of our benchmark curation, as well as your constructive comments on the benchmark design, analysis, and baseline comparisons. We sincerely hope that our response addresses your concerns.

W1: Potential Bias Toward Unnecessary Asking.

Thank you for raising the concern that our evaluation protocol may encourage models to ask unnecessary questions. Our protocol was motivated by a limitation of ProactiveBench. In that work, the prompts do not explicitly inform models that they can ask questions, resulting in near-zero asking rates across models, including GPT models. We believe this approach does not adequately measure models’ ability to ask questions proactively. Therefore, in our setting, the prompt explicitly states that information may or may not be missing and allows the model to choose whether to ask. This formulation is intended to provide a more objective assessment of proactive asking.

To investigate whether our protocol introduces a bias toward asking, we evaluated gpt-4o on the PO proactive subset of StreamingBench, using StreamHarness and our existing evaluation method, with only a minor modification allowing the model to answer directly without first issuing an ASK. This subset contains no missing information and therefore requires no asking. As shown below, the unnecessary ask rate was 0.00%, indicating that our protocol does not induce unnecessary asking. Meanwhile, accuracy increased from 56.86% to 70.59%, further supporting our protocol’s ability to more accurately assess offline models’ proactive understanding capabilities.

SettingAccUnnecessary Ask Rate ↓
Raw setting56.86%N/A
Ours70.59%0.00%

W2: Reliability of Analysis Using Gemini.

Thank you for highlighting this limitation. We extended the oracle ablation to GPT-5.6-Sol and Qwen-3.8 Flash under the same settings. As shown in the table below, WHEN_ASK and WHEN_WHAT_ASK improve Ask-F1 over NONE by 17.73 and 40.64 percentage points for GPT-5.6-Sol, and 24.43 and 46.29 points for Qwen-3.8 Flash, respectively. Both models show the same main trend as Gemini-3.8 Flash, supporting when and what to ask as major bottlenecks across the evaluated models.

ModelNoneWHEN_ANSWHAT_ASKWHEN_ASKWHEN_WHAT_ASKALL
Gemini-3.8-flash37.1739.4040.9658.30(+21.13)81.0(+43.83)83.3
gpt-5.6-sol18.1020.3124.9535.83(+17.73)58.74(+40.64)61.22
Qwen-3.8 Flash21.0325.8822.1745.46(+24.43)67.32(+46.29)70.18

W3: Lack of a Pure Text Baseline.

Thank you for highlighting the missing text-only baseline. During benchmark construction, human annotators verified all eight subtasks under two conditions: without and with the requested evidence (text, images, or video). Even with access to the original video, the answer could not be uniquely determined without the additional evidence, whereas providing it resolved the uncertainty (Section A.7.2).

To directly assess textual shortcuts, we additionally evaluate Gemini-3.8-flash using only the question and candidate options, independently for the ASK and ANSWER stages. We report accuracy under three settings: original option order, shuffled order, and shuffled order with text rephrased by Qwen3.8-flash. As shown in the table below, both ASK and ANSWER accuracies remain close to the random-selection baseline across all three settings. These results suggest that textual cues alone provide limited predictive value.

SettingASK Acc.ANSWER Acc.
Random selection19.5621.48
Original order22.4823.13
Shuffled order20.6323.77
Shuffled order & Text rephrased18.9222.70
AuthorsOpenReview ↗

Response to Reviewer p5c4

Q1: Analysis of the Poor Performance of Streaming Models.

Thank you for the question. Existing streaming models are mainly trained with special tokens such as <wait> and <response> to represent silence and responses, respectively. For a fair comparison, we map <wait> to WAIT and <response> to ASK or ANSWER. We then use regular expressions to determine whether the generated response corresponds to ASK or ANSWER. The errors of Instruction following are rare (2.3% in both settings). The dominant failure mode is remaining silent throughout the entire turn (75.6%). This suggests that current models may not have fully learned the interaction policies needed for proactive streaming tasks. One possible explanation is the limited generalizability of existing training methods or datasets.

A simple improvement would be to introduce an <Ask> token into the model vocabulary and provide corresponding training examples. More broadly, future work could develop training strategies that help models generalize their decisions about when to act across a wider range of interactions, beyond ASK and ANSWER.

Q2: Clarification of the Typos in Figure 3.

Thank you for pointing this out. We apologize for the typo in Figure 3: “The branch is Cuisinart” should read “The brand is Cuisinart.” This typo was introduced by us during manual figure preparation. We have already corrected this error in Figure 3.

We thank the reviewer again for the thoughtful and constructive feedback, which has helped us improve the clarity and rigor of our work.

AuthorsOpenReview ↗

Response to Reviewer p5c4

Dear Reviewer p5c4,

Thanks for your time and effort in reviewing this paper. Your comments are valuable to improve the quality of this paper. As the author-reviewer discussion period will end in about 2 days, we gently request that you read the rebuttal and check whether your concerns are addressed. Thanks again for your time and help in improving the quality of this paper.

Regards from the authors.

Reviewer_enhiOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AskStream, a benchmark for proactive information seeking in streaming video understanding. Beyond WAIT and ANSWER, it requires a model to decide when and what to ASK when visual evidence or user-provided information is insufficient. The benchmark includes eight subtasks covering missing visual evidence, personalized concepts, ambiguous or incorrect user queries, and missing or outdated user preferences. It contains 438 videos and 1,401 questions, with manually annotated temporal action windows.

The authors propose Ask-F1 to jointly assess the timing and correctness of ASK and ANSWER, and StreamHarness to supply hierarchical video memory to offline MLLMs. Evaluations are conducted in multiple-choice and open-ended settings; the highest reported multiple-choice Ask-F1 is 52.18%.

Critical strengths
  1. Meaningful problem formulation. The paper addresses an important and relatively underexplored interaction capability. Introducing ASK into streaming video understanding is a meaningful extension of existing WAIT/ANSWER-style proactive benchmarks, and the taxonomy covers multiple sources of uncertainty from both videos and users.
  2. Substantial benchmark construction and validation. The benchmark includes fine-grained temporal action windows and both multiple-choice and open-ended evaluations. The authors further report independent temporal re-annotation (Appendix A.7.3) and human agreement with the open-ended judge (Appendix A.5), which strengthen confidence in the annotation effort.
Critical weaknesses
  1. Insufficient evidence for the benefit of proactive asking. Figure 2 shows that revealing missing visual evidence improves accuracy, but does not establish the advantage of model-driven ASK over simple information-request strategies. Moreover, the benchmark lacks cases where no ASK is needed, making it difficult to evaluate whether models can avoid unnecessary questions.
  2. Potential issues in benchmark construction and evaluation. In UPM/UPC, the correct ASK depends on a randomly selected target entity, potentially penalizing other reasonable clarification questions. Multiple-choice options may also introduce textual shortcuts. Furthermore, the proposed Ask-F1 does not directly measure end-to-end task success, and its aggregation may obscure substantial differences across subtasks.
Questions for authors

Q1. Can models decide when not to ask? Human verification ensures that no instance can be answered without the missing information. Every turn therefore requires an ASK, and unnecessary asking is never penalized. Could the authors add negative cases where the available evidence already suffices and report unnecessary-ASK (false-positive) rates, to test whether models can decide not to ask? Q2. Do multiple-choice candidates leak the answer? Scores drop sharply from the multiple-choice to the open-ended setting (Gemini-3.8 Flash: 52.18 → 29.19 overall; 81.36 → 18.43 on UPC), suggesting that the candidates provide substantial cues. Since all candidates are generated by GPT-5.6 Sol, the correct option may also be identifiable through stylistic artifacts. Could the authors report a text-only (no-frame) baseline, test option shuffling and paraphrased candidates, and report the rank correlation of models between the two settings? Q3. Can a preference ASK identify a randomly designated target? In UPM/UPC, the preferred item is randomly designated. Yet the judge requires the ASK to target a dimension on which that item's value is unique. How can a model satisfy this before knowing the user's preference? Would other reasonable clarification questions be accepted, and would the simulated user answer them? Gemini-3.8 Flash scores 64.03 on UPM in the multiple-choice setting but only 14.41 in the open-ended setting, which suggests this criterion may drive the results. Could the authors report human agreement on ASK validity, or accept multiple valid ASK strategies? Q4. Does pooled Ask-F1 bias the model rankings? Global Ask-F1 pools S_ask and S_ans over all turns (Gemini-3.8 Flash: 44.63 and 62.80 → 52.18), which implicitly weights subtasks by sample count. Its macro average over the eight subtasks is only 40.66. Macro-averaging also reshuffles the rankings: Kimi K2.6 rises from 6th to 2nd, MiMo-V2.5 from 7th to 4th, and GPT-5.6-Sol falls from 3rd to 5th. Could the authors report per-subtask sample counts, macro-averaged Ask-F1, end-to-end turn success rates, and confidence intervals grouped by video?

3 comments
AuthorsOpenReview ↗

Response to Reviewer enhi

Thank you for your careful and insightful reviews. We appreciate your recognition of the meaningful&important extension of the proactive streaming paradigm to incorporate ASK and the substantial benchmark construction&validation, as well as your constructive feedback on the benefits of proactive asking, the benchmark design and evaluation. We sincerely hope that our response addresses your concerns.

W1:Insufficient evidence for the benefit of proactive asking

Regarding whether model-driven asking outperforms simple information-request strategies, we would like to clarify that model-driven asking is an interaction paradigm also adopted by prior work, including AskBench and ProactiveBench. Our work follows this paradigm and focuses on establishing a benchmark for proactive asking in streaming video understanding, evaluating models’ ability to determine when and what to ask.

Q1: The evaluation framework may bias models toward unnecessary ASK.

Thank you for highlighting this concern. We agree that recognizing when no further questions are needed is an important aspect of proactive asking. Our system prompt explicitly states: “The current question or the video may or may not contain missing information.” This formulation asks the model to judge whether essential information is missing, even though all AskStream instances involve missing information. The intention is to assess models’ proactive asking capabilities more objectively by requiring them to identify the information gap themselves.

To examine whether our evaluation introduces a bias toward asking, we evaluated gpt-4o on the PO proactive subset of StreamingBench, which contains no missing information and therefore no need to ask questions. We used StreamHarness and our existing evaluation method, with only a minor modification allowing the model to answer directly without first issuing an ASK. As shown below, the unnecessary ask rate was 0.00%, while accuracy increased from 56.86% to 70.59%. These results provide evidence that our evaluation does not induce unnecessary asking in this setting. The accompanying accuracy gain also indicates that our setup better supports offline models’ performance on proactive video understanding tasks.

SettingAccuracyUnnecessary Ask Rate ↓
Original setting56.86%N/A
Ours70.59%0.00%

Q2: Multiple-choice options may leak answers.

Thank you for pointing out this weakness. We agree that candidate options may provide useful cues. To directly assess this possibility, we additionally evaluate Gemini-3.8 Flash using only the question and candidate options, independently for the ASK and ANSWER stages. We compare three settings: original option order, shuffled order, and shuffled order with candidate options rephrased by a different model, Qwen-3.8 Flash. As shown below, text-only accuracies remain near chance, suggesting limited textual shortcuts. Human verification further confirms that additional evidence or clarification is needed to uniquely determine the answer (Section A.7.2). Across the seven offline models, overall Ask-F1 rankings correlate at Spearman’s ρ = 0.8214 between the multiple-choice and open-ended settings.

SettingASK Acc.ANSWER Acc.
Random selection19.5621.48
Original order22.4823.13
Shuffled order20.6323.77
Shuffled order + rephrased options18.9222.70
AuthorsOpenReview ↗

Response to Reviewer enhi

Q3: How models can ASK without knowing users’ preferences in UPM/UPC?

Thank you for pointing out this ambiguity in our description. During the construction of the UPM/UPC subtasks, we manually crop images of all candidate entities to provide visual evidence for the judge. In the open-ended setting, we iterate over all entities observed up to the model’s ASK timestamp, designating each entity as the “Correct Image” in a separate judge. We accept the ASK if any judges it valid. This procedure avoids tying ASK validity to a single target.

Regarding simulated-user feedback, we acknowledge a limitation of the current implementation. Open-ended ASKs may concern preference dimensions that we cannot fully anticipate during annotation. Consequently, the simulator returns the same pre-annotated preference information for every accepted ASK. This information makes the original task answerable, but does not necessarily match the preference dimension requested by the model. We appreciate the reviewer’s observation and have incorporated these clarifications into the paper to make the evaluation protocol and limitations more transparent.

Regarding agreement between ASK judgments and human assessments, Appendix A.5 and Table 4 report validation on 300 randomly sampled turns from the open-ended evaluation. Two human annotators independently assessed ASK validity, with disagreements resolved by a third annotator. Human–human agreement on ASK validity was 95.33%, while agreement between the judge and human consensus was 92.67%.

Q4: The Ask-F1 may affect the model ranking.

Thank you for highlighting this aggregation sensitivity. We agree that pooled Ask-F1 alone does not adequately capture performance consistency across the eight subtasks. The tables below report per-subtask turn counts, pooled Ask-F1, macro-averaged Ask-F1, and end-to-end success rates in the multiple-choice setting, with 95% confidence intervals estimated using a video-cluster bootstrap with 10,000 resamples drawn with replacement. Gemini remains first across all three metrics, while the relative rankings of other models change. We have added the corresponding statistics for both the multiple-choice and open-ended settings to the paper.

KVEMVIMICMPACMAUQWUQUPMUPC
17131614950107219174215
Modelpooled Ask-F1 (%)Macro Ask-F1 (%)End-to-End Success Rate (%)
GPT-5.6-Sol20.50 [18.85, 21.98]12.80 [11.35, 13.96]20.99 [17.59, 24.52]
Claude Sonnet 522.48 [20.36, 24.54]16.49 [14.30, 18.18]19.20 [16.01, 22.62]
Gemini-3.8 Flash52.18 [49.42, 54.83]40.66 [38.31, 42.60]46.47 [41.49, 51.53]
MiMo-V2.514.66 [12.69, 16.68]15.82 [14.12, 17.41]19.99 [17.22, 22.85]
Kimi K2.616.52 [14.21, 18.89]17.01 [15.25, 18.40]11.56 [9.65, 13.60]
Qwen3.8-Flash17.60 [15.01, 20.01]10.27 [7.95, 12.02]9.21 [7.38, 11.18]
DeepSeek-V4.1-Flash19.69 [17.88, 21.16]10.71 [9.09, 11.95]22.20 [18.09, 26.37]
MOSS-VL-Realtime0.00 [0.00, 0.00]0.00 [0.00, 0.00]0.00 [0.00, 0.00]
MMDuet20.00 [0.00, 0.00]0.00 [0.00, 0.00]0.00 [0.00, 0.00]
VideoLLM-online0.00 [0.00, 0.00]0.00 [0.00, 0.00]0.00 [0.00, 0.00]
MiniCPM-o 4.50.12 [0.00, 0.32]0.84 [0.00, 1.35]0.14 [0.00, 0.37]
TimeChat-Online4.18 [2.56, 5.77]1.68 [1.05, 2.28]1.57 [0.90, 2.31]
JoyAI-VL-Interaction4.91 [3.23, 6.28]1.22 [0.81, 1.68]2.07 [1.19, 3.05]
AURA7.22 [5.56, 8.67]2.61 [1.87, 3.31]2.93 [2.06, 3.89]

We thank the reviewer again for the thoughtful and constructive feedback, which has helped us improve the clarity and rigor of our work.

AuthorsOpenReview ↗

Response to Reviewer enhi

Dear Reviewer enhi, thank you again for your thoughtful feedback. If you have a chance before the discussion closes, we’d appreciate hearing whether the additional experiments on unnecessary asking and textual shortcuts, along with the new evaluation metrics, address your concerns. If they do, we would be grateful if you would consider a higher rating.

Reviewer_HbNfOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper introduces AskStream, a benchmark for streaming video understanding under uncertainty arising from either the video content or the user query. The authors argue that existing streaming video benchmarks assume that the information required to answer a query is already present in the video. AskStream is a multi-turn benchmark designed to probe whether streaming models can request the information needed to answer a query. It comprises four task categories and eight subtasks, with 438 videos and 1,401 questions. The paper evaluates proprietary VLMs, open-source VLMs, and streaming video models, and introduces a task formulation and a metric (Ask-F1) that measure both how well a model asks for missing information and how well it uses that information to answer the query.

Critical strengths
  • The problem setting is interesting and well motivated by how video streams are used in practice. Enabling streaming models to request unavailable information before answering is a sound problem statement.
  • The benchmark design is comprehensive. It frames streaming video understanding as a simulated environment in which the model can take actions to gather information, supported by an extensive annotation pipeline that constructs these interactions from existing videos.
Critical weaknesses
  • My primary concern is with the main results in Table 2. The authors state that, for stronger offline models, "SaskS_{ask} is significantly lower than SansS_{ans}" (lines 403–404), and conclude that ASK is the bottleneck. However, this does not hold for several open-source models in the multiple-choice setting, e.g., Kimi K2.6 (SaskS_{ask} = 19.35 vs. SansS_{ans} = 14.41) and Qwen-3.8 Flash (31.75 vs. 12.17). In addition, the oracle ablation in Figure 6 shows that providing "when to ASK" yields by far the largest gain. Together, these results suggest that the conclusion may be driven by the strict timing constraint in SaskS_{ask} rather than by the models' ability to ask.
    • Could the authors justify the strict temporal window for the ASK action? How do models perform if the window is relaxed to Wiask=[qi,ei)W^{ask}_i = [q_i, e_i), i.e., any ASK issued before the end of the answer window is accepted? This would show model performance without the timing constraint on ASK.
    • In Figure 6(b), the oracle for what to ask alone (WHAT_ASK) improves Ask-F1 by only 3.79 points, whereas adding the same oracle on top of the "when" oracle improves it by 22.7 points (WHEN_WHAT_ASK vs. WHEN_ASK: 81.0 − 58.3). This suggests that correct ASK content is largely masked by the timing constraint, i.e., the gap reflects the evaluation metric rather than the models' underlying capability.
  • For the specialized streaming models, the lack of training for this behavior is a likely explanation for the near-zero results. As it stands, the analysis offers little insight into how these models perform on the task.
Questions for authors
  • Could the authors explain why the WHEN_ASK oracle yields such a large gain?
  • Some examples in Figure 3 are unclear or contain errors. For instance, the KVEM query reads "When she finished speaking, told me the brand of the fourth object from the left", while the corresponding ASK is "clarify the currently highest entity", which refers to the target entity differently from the query; the answer also reads "The branch is Cuisinart". Similarly, the VIM query reads "When I picked up the yellow crowbar, it told me the color of the object hanging behind me". Errors of this kind in the showcased examples raise concerns about the quality of the curated benchmark questions.
11 comments
AuthorsOpenReview ↗

Response to Reviewer HbNf

Thank you for your careful and insightful reviews. We appreciate your recognition of the practical motivation for introducing ASK in real-world streaming video scenarios and comprehensive benchmark design, as well as your insightful comments on the evaluation protocol, experimental results, and benchmark quality. We sincerely hope that our response addresses your concerns.

W1: Overgeneralization from Partial Experimental Results.

Thank you for highlighting this ambiguity. We agree that “for stronger models” was insufficiently precise. Our intended observation concerns Gemini-3.8 Flash, the best-performing model in our evaluation, whose S_ask (44.63%) is substantially lower than S_ans (62.80%). We have revised the conclusion in the paper to state that ASK remains a limitation for our benchmark's SOTA model.

W2: Potential Confounding Effect of the Strict Timing Windows.

Thank you for raising these concerns about the strict ASK timing windows. The strict ASK windows reflect two information acquisition scenarios. In Type I (wait before asking), the initial observations are insufficient to identify the uncertainty, so the model must wait for relevant evidence before issuing a grounded ASK. Premature requests may lack sufficient observational support. In Type II (ask before waiting), the uncertainty is already identifiable from the initial observations. Asking before the ANSWER window allows clarification to proceed while the model waits for the target event, reducing potential delays once the conditions for answering are met.

To examine the effect of this restriction, we evaluated Gemini-3.8 Flash in the multiple-choice setting on KVEM and VIM (Type I), and UPM and UPC (Type II), comparing end-to-end turn success rates under the strict ASK windows and relaxed windows of ([q_i,e_i)). As shown below, the gains are small for Type I and somewhat larger for Type II. Our analysis of model error types explains the above findings. Wrong Ask accounts for 81% of Type I failures under the relaxed ASK windows. Moreover, 96% of the cases previously classified as Early Ask under the strict windows are reclassified as Wrong Ask under the relaxed windows, indicating that these premature requests still exhibit hallucinations even when their timing is accepted. Meanwhile, 72% of Type II failures are attributable to Wrong Ask. This is because Wrong Ask was already the predominant error type under the strict time window (60.00%/65.45%), whereas Miss Ask accounted for only 12.94%/18.18%. Consequently, relaxing the time window enables correct asks in only the cases previously classified as Miss Ask.

Regarding Figure 6(b), we agree that temporal validity can mask the benefit of correct ASK content. Because Ask-F1 credits ASK content only when the timing requirement is satisfied, the 3.79-point gain from WHAT_ASK alone can underestimate its contribution. With oracle ASK timing, adding the content oracle improves Ask-F1 from 58.30 to 81.00 (+22.70 points), showing that What to ask remains an important bottleneck after timing is controlled. These gains reflect an interaction between timing and content and should not be interpreted as independent measures of the model’s underlying capabilities.

All of the above experimental results indicate that When and What to ask are the bottlenecks in the model's performance. We are extremely grateful to the reviewer for pointing out the inappropriate parts in our analysis and for the insightful suggestions. We have corrected the conclusions of the analysis in the paper.

SettingKVEMVIMUPMUPC
Strict ASK window38.606.9651.1574.42
Relaxed ASK window39.77 (+1.17)7.91(+0.95)56.32 (+5.17)79.07(+4.65)

W3: Insufficient Analysis of Streaming Models.

Thanks for the reviewer for pointing out the insufficient analysis of streaming models. Existing streaming models are trained with special tokens such as <wait> and <response> to represent silence and responses, respectively. We find that streaming models primarily fail by remaining silent throughout the entire turn, accounting for 75.6% of their failures in both settings. This suggests that the policies required for proactive streaming interaction may not yet have been sufficiently learned by existing streaming models. This may be because exist training methods, or training datasets have limited generalizability. A straightforward improvement would be to add an <Ask> token to the model vocabulary and train the model with corresponding data. Alternatively, We should investigate a more general training strategy to improve models’ ability to determine when to act, extending beyond ASK and ANSWER.

AuthorsOpenReview ↗

Response to Reviewer HbNf

Q1: Explaining the Large Gain from the WHEN_ASK Oracle

Thank you for raising this question. The substantial gain from the WHEN_ASK oracle is mainly attributable to the role of ASK timing as a prerequisite in the sequential interaction pipeline. First, providing WHEN_ASK allows cases that previously failed due to incorrect ASK timing to proceed to subsequent stages. Second, in some of these cases, the model is already capable of answering correctly once the necessary information is provided. However, incorrect ASK timing excludes these cases from the subsequent ANSWER evaluation. By correcting the timing, WHEN_ASK brings these cases into the evaluation, allowing the model’s existing answering capability to be reflected in the final score.

Q2: Potential Annotation Inconsistencies in the Figure 3.

Thank you for pointing out the Typos. We apologize for these typing errors, which were introduced when we manually added the text to Figure 3. Specifically, “The branch is Cuisinart” should read “The brand is Cuisinart,” and the VIM query should read “When I pick up the yellow crowbar, tell me the color of the object hanging behind me.” We have already corrected both errors in Figure 3.

Regarding the inconsistency between “the fourth object from the left” in the query and “clarify the currently highest entity” in the model’s ASK, thank you for pointing out the ambiguity in our description. This is part of our dataset design to avoid textual priors and model hacking, rather than an error. First, both spatial descriptions refer to the juicer, satisfying the KVEM requirement that the model request the correct entity. Second, your main question may be why the model does not directly request “the fourth object from the left,” but instead uses another spatial description to locate the target. This is because, if the model could directly request “the third object” or “the fourth object,” it might exploit textual shortcuts without using visual information. The model cannot directly reuse the referring expression in the query. Instead, it must provide another spatial description based on the video. Specifically, in the multiple-choice setting, we provide a spatial description for each masked object, and the model must select the one corresponding to the correct entity. In the open-ended setting, we explicitly instruct the model through the prompt to locate the required entity using spatial information from the video, without directly reusing the object reference in the query. Responses that directly reuse that reference are judged incorrect. These settings are intended to ensure that the model actually uses the video information. We hope this response addresses your concerns about the quality of the dataset.

We thank the reviewer again for the thoughtful and constructive feedback, which has helped us improve the clarity and rigor of our work.

AuthorsOpenReview ↗

Response to Reviewer HbNf

Dear Reviewer HbNf,

Thanks for your time and effort in reviewing this paper. Your comments are valuable to improve the quality of this paper. As the author-reviewer discussion period will end in about 2 days, we gently request that you read the rebuttal and check whether your concerns are addressed. Thanks again for your time and help in improving the quality of this paper.

Regards from the authors.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary for Submission 16863

Dear Area Chairs and Reviewers,

We sincerely thank all reviewers for their careful assessment and constructive feedback on our submission. We are particularly grateful for the recognition of the interesting and practically well-motivated problem setting and comprehensive benchmark design (Reviewer HbNf), the meaningful formulation and substantial benchmark construction and validation (Reviewer enhi), and the novelty, real-world importance, solid evaluation, and rigorous curation process (Reviewer p5c4).

We also appreciate that, following our additional experiments and clarifications, Reviewers HbNf and p5c4 indicated that their concerns had been addressed and increased their ratings. The reviewers rate the paper 3 (Weak acceptance), 3 (Weak acceptance), and 3 (Weak acceptance), respectively.

Summary of Updates

All concerns raised by the reviewers have been addressed point-by-point, including those related to ASK timing, oracle evaluation, unnecessary asking, textual shortcuts, preference-query validity, metric aggregation, streaming-model failures, benchmark quality, and presentation. We have completed the following additional experiments and analyses and will incorporate them into the revised manuscript:

  • ASK timing. Relaxing the ASK window yields only modest gains on KVEM/VIM (+1.17/+0.95) and moderate gains on UPM/UPC (+5.17/+4.65). Under relaxed timing, Wrong ASK accounts for 81%/72% of Type-I/II failures, and 96% of strict-window Early ASKs become Wrong ASKs, confirming that both timing and content are bottlenecks.

  • Oracle validation. With correct timing, adding correct ASK content raises Gemini’s Ask-F1 from 58.30% to 81.00%. Cross-model experiments show the same pattern: WHEN_ASK and WHEN_WHAT_ASK improve GPT-5.6-Sol by 17.73/40.64 points and Qwen-3.8 Flash by 24.43/46.29 points.

  • Unnecessary ASK control. On the no-missing-information subset of StreamingBench, our protocol achieves a 0.00% unnecessary-ASK rate while improving GPT-4o accuracy from 56.86% to 70.59%, indicating that permitting ASK does not inherently encourage unnecessary questions.

  • Textual-shortcut controls. Text-only performance remains near chance under original, shuffled, and independently paraphrased options; multiple-choice and open-ended model rankings also correlate strongly (Spearman’s ρ = 0.8214). These results suggest limited leakage from option order or wording.

  • Expanded metrics. We additionally report per-subtask counts, macro Ask-F1, end-to-end success, and video-clustered 95% confidence intervals for both evaluation settings. Gemini remains first across all aggregate metrics, while ranking changes among other models are made transparent.

  • Preference ASK validity. UPM/UPC ASKs are accepted if valid for any entity visible at the ASK timestamp, rather than one predetermined target. Human–human and judge–human ASK agreement reach 95.33% and 92.67%, respectively; we also document the limitation of using pre-annotated simulator feedback.

  • Streaming model and presentation analysis. Streaming models mainly fail by remaining silent (75.6%), whereas instruction-following errors are rare (2.3%), pointing to deficiencies in learned interaction policies. We also corrected Figure 3, clarified the KVEM referring-expression design and oracle setup, and revised the conclusions to reflect the joint importance of when and what to ask.

We sincerely appreciate the reviewers’ valuable feedback. It led to substantially stronger controls, a more complete evaluation protocol, clearer interpretations of ASK timing and content, and a more transparent discussion of the benchmark’s capabilities and limitations.

Sincerely,

Authors of Submission 16863

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.