- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper explores the problem of proactive interaction in streaming video understanding, where models must incrementally process incoming audio-visual content and autonomously decide whether and when to respond without explicit user queries. To formalize this, the authors define the "Proactive Instruction Following" (PIF) task and introduce PIFBench, a benchmark comprising 1,993 test samples that cover both visual and audio information while supporting stable rule-based verification. To improve model capabilities, the authors constructed a structured training dataset of 8K samples named Proactive-Instruct-8K. Furthermore, they fine-tuned the Qwen3-Omni-30B-A3B-Instruct model to develop Qwen3-Omni-Proactive by introducing a special <Silence> token to the vocabulary and utilizing a Chain-of-Thought (CoT) augmented streaming format. Experimental validations demonstrate that while current mainstream models have limited proactive interaction capabilities, the proposed Qwen3-Omni-Proactive model improved the accuracy on PIFBench from 13.79 to 19.42 under the same online evaluation setting.
Critical strengths
(1) Streaming video understanding is a critical research direction with great application potential, and benchmarks for this field are valuable.
(2) The benchmark contains 1,993 samples covering not only visual-only and audio-visual multimodal tasks but also short-range and long-range visual/audio dependencies. The dataset also introduces 10% negative samples specifically designed to test the models' ability to remain silent throughout the entire video. This addresses the issues of previous benchmarks that omitted audio or relied on laborious open-ended evaluation.
(3) The authors also contribute Proactive-Instruct-8K for training and train the Qwen3-Omni-Proactive model, which can be useful for the community.
Critical weaknesses
(1) While introducing PIFBench is a primary contribution of this paper, Section 3.3 significantly oversimplifies the data construction process. Crucial methodological details such as negative sample generation, response format variants, and human verification standards are entirely relegated to the appendices. This excessive reliance on the appendix undermines the standalone completeness of the main text.
(2) Both the Proactive-Instruct-8K and PIFBench heavily rely on multimodal models for automatic annotation and quality filtering, which might introduce noisy or imperfect supervision signals.
(3) Given the focus on streaming video and real-time proactive interaction, the paper lacks a quantitative analysis of memory usage, inference latency, or throughput.
(4) The use of a <Silence> token is an incremental tweak that mimics existing paradigms, such as EOS prediction in VideoLLM-Online. More critically, forcing the model to autoregressively generate CoT reasoning at every temporal chunk to balance training labels introduces severe inference latency. This design trades essential real-time throughput for label balance, fundamentally contradicting the latency-sensitive requirements of streaming applications and rendering the approach highly impractical for real-world deployment.
(5) Lack of discussion of recent papers on streaming proactive video understanding [1] - [3]
[1] StreamReady: Learning What to Answer and When in Long Streaming Videos, 2026 CVPR
[2] StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistant, 2025 NIPS
[3] Streaming Video Instruction Tuning, 2026 CVPR
Questions for authors
Please see the weakness.
4 comments
Looking forward to Feedback as Discussion Deadline Approaches
Hi, we sincerely thank you very much for these constructive comments and evaluation of our manuscript. As the discussion phase will be closed on November 18 11:59pm AoE, we would like to kindly ask you to take a look at our responses and reevaluate our work based on our clarifications. Please let us know whether our response addresses your concerns or whether there is any further detail we can provide to help address these concerns.
Thank you again for dedicating your time to reviewing our paper.
Thank you for the revisions. W1 and W5 are resolved, and I appreciate the new CoT comparison. Two concerns remain. The annotation quality of Proactive-Instruct-8K is still unquantified, and the aggregate runtime in Table 6 leaves per-chunk latency in streaming use unclear. I therefore maintain my rating of 2.
Response to Revierwer 77ta
Dear Reviewer 77ta,
Thank you for your feedback. We’re glad our earlier response addressed some of your concerns. We’d like to clarify the two remaining points.
1. Annotation quality of Proactive-Instruct-8K
To measure annotation quality, we built a simple agent loop to check examples for errors and hallucinations and to try to fix them. The prompt is presented in Appendix J. Given the time and cost limits, we have checked about 300 examples so far. We used a 0–10 scoring scale and obtained the following results.
| Score | Meaning | Share |
|---|---|---|
| 0–3 | Contains severe hallucinations | 7.7% |
| 4–6 | The instruction does not match the task well | 19.7% |
| 7–9 | Usable and high quality | 70.7% |
| 10 | Meets all quality criteria | 2.0% |
After a simple correction pass, 47.8% of the examples that initially scored 0–3 were rescored in the 7–9 range. This initial check suggests that most of the examples we reviewed are usable and of good quality. As shown in our paper, we have observed higher evaluation scores after training on Proactive-Instruct-8K, which supports its usefulness as training data.
Manually reviewing and correcting every example can be costly for us. Our pipeline makes it possible to generate this type of data at scale and at low cost. We will release the dataset publicly so other researchers can use it and build on it.
2. Per-chunk latency in streaming
Latency depends on the GPU hardware and the inference engine, so we compare all models under the same GPU setup. Different examples can produce different chunks, and the chunks can vary even when the same model processes the same example more than once. We therefore report the average end-to-end runtime across examples, including model loading and other overhead. Averaging over multiple examples helps reduce the effect of fluctuations in individual runs.
We know the discussion deadline is approaching, but we’d still like to do our best to address any remaining questions or concerns. Your feedback helps us make our work more solid. Thank you again, and we hope you’ll have a chance to read our response and share your thoughts before the discussion closes.
Response to Reviewer 77ta
Dear Reviewer 77ta,
Thank you for your careful reading of our paper and for your constructive feedback. We respond to your comments point by point below.
W1: We thank the reviewer for raising this point. In the initial submission, we aimed to make the core construction process self-contained by describing the data sources, task taxonomy, and task-specific pipelines in Section 3.3 and summarizing them in Figure 2. To meet the page limit, we placed supplementary procedures for negative samples, response format variants, and benchmark conversion in the appendix. These components increase the benchmark’s diversity. Section 3.3 describes the essential steps of data construction. With the additional page in the revised version, we have added descriptions of negative sample generation, response format variants, and benchmark conversion to Section 3.3, while retaining fine-grained procedural details in the appendix for conciseness.
W2: We thank the reviewer for raising this concern. For PIFBench, multimodal models generate candidate annotations, which are then reviewed, corrected, or removed by human annotators according to explicit criteria. For Proactive-Instruct-8K, we agree that automatic annotation and filtering may leave some imperfect supervision. We use this approach because training data must be scalable, and verifying every sample manually is impractical. Our fine-grained pipeline decomposes annotation into constrained steps that are more manageable for multimodal models, helping us balance scalability and data quality. The improvement over the base model suggests that Proactive-Instruct-8K provides useful supervision, although it does not imply that all annotations are error free. We will release the dataset to support community inspection and refinement, and will use feedback to inform future improvements.
W3: Our method does not modify the model architecture or add inference modules, it only adds a special token to indicate when the model should remain silent. Thus, the memory required to store the model parameters remains unchanged. Since the base model does not support proactive interaction, directly comparing its efficiency with the adapted model would conflate the cost of enabling proactive interaction with the cost of our training strategy. We therefore added a controlled comparison between the CoT and non-CoT variants in Section 5.3.3. Under the same evaluation setting, we report their overall inference runtime and response length to characterize the efficiency tradeoff introduced by CoT. We thank the reviewer for encouraging us to include this analysis.
W4: We do not present the <Silence> token as a novel contribution in Section 4.2. Our main goal is to use PIFBench to evaluate proactive interaction and, since this capability is not explicitly supported by general-purpose models, to improve it through training data and establish a baseline. The token is a simple mechanism for expressing the silence decision. We agree that generating CoT at every chunk trades inference efficiency for accuracy. As noted in Section 5.3.3, we recommend the CoT-augmented variant because it provides richer supervision and achieves better overall performance. However, explicit CoT need not be retained in every deployment setting. Users can shorten or omit it when lower latency is more important, while accepting a possible reduction in accuracy. We have revised the wording in Section 5.3.3 to clarify this tradeoff and added: 'For applications that prioritize lower runtime, the non-CoT variant may be preferable.' We thank the reviewer for raising this important concern.
W5: We were aware of these works when conducting our study, but the initial related-work discussion was selective due to space constraints. In the revised version, with the additional page, we have added discussions of StreamReady, StreamBridge, and Streaming Video Instruction Tuning. We describe the first two as method-focused approaches and the third as work on streaming video instruction data, and clarify their relationship to our work. We thank the reviewer for drawing our attention to these relevant papers and helping us improve the discussion.
We hope that our revisions and clarifications address your concerns. Thank you again for your thoughtful feedback, which helped us improve the paper.
Sincerely,
The Authors