acceptodds
Under review as a conference paper at ICLR 2027

PIFBench: Evaluating and Improving Proactive Interaction in Streaming Video Understanding

Abstract

As AI systems move toward real-time human-AI interaction, evaluating models' ability to understand continuously arriving visual and audio information becomes increasingly important. In this context, streaming video understanding requires models to process incoming content incrementally and decide whether and when to respond. The ability to make this decision without an explicit user query is known as proactive interaction. Existing benchmarks for proactive interaction often omit audio or rely on laborious open-ended evaluation, and effective methods for improving this capability in omni-modal models remain underexplored. To address these gaps, we study proactive interaction from a full-stack perspective, covering evaluation, data construction, and model adaptation. For evaluation, we formalize Proactive Instruction Following (PIF), a task defined by providing an instruction before a video begins and requiring the model to process the video incrementally, remain silent when the specified condition is unmet, and produce the required response when it is met. Based on this task, we introduce PIFBench, a comprehensive benchmark for evaluating proactive interaction in streaming video understanding. PIFBench contains 1,993 items, covers both visual and audio information, and supports stable rule-based verification. Our evaluation shows that mainstream models have limited proactive interaction capability. Thus, we build Proactive-Instruct-8K, a structured training dataset for omni-modal models. For model adaptation, we make reusable modifications to the training framework and fine-tune Qwen3-Omni-30B-A3B-Instruct on Proactive-Instruct-8K, yielding Qwen3-Omni-Proactive, which improves accuracy on PIFBench from 13.79 to 19.42 under the same online evaluation setting.

Primary area:
datasets and benchmarks
Keywords:
Multimodal Large Language Models, Streaming Video Understanding, Benchmarking

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_77taOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper explores the problem of proactive interaction in streaming video understanding, where models must incrementally process incoming audio-visual content and autonomously decide whether and when to respond without explicit user queries. To formalize this, the authors define the "Proactive Instruction Following" (PIF) task and introduce PIFBench, a benchmark comprising 1,993 test samples that cover both visual and audio information while supporting stable rule-based verification. To improve model capabilities, the authors constructed a structured training dataset of 8K samples named Proactive-Instruct-8K. Furthermore, they fine-tuned the Qwen3-Omni-30B-A3B-Instruct model to develop Qwen3-Omni-Proactive by introducing a special <Silence> token to the vocabulary and utilizing a Chain-of-Thought (CoT) augmented streaming format. Experimental validations demonstrate that while current mainstream models have limited proactive interaction capabilities, the proposed Qwen3-Omni-Proactive model improved the accuracy on PIFBench from 13.79 to 19.42 under the same online evaluation setting.

Critical strengths

(1) Streaming video understanding is a critical research direction with great application potential, and benchmarks for this field are valuable.

(2) The benchmark contains 1,993 samples covering not only visual-only and audio-visual multimodal tasks but also short-range and long-range visual/audio dependencies. The dataset also introduces 10% negative samples specifically designed to test the models' ability to remain silent throughout the entire video. This addresses the issues of previous benchmarks that omitted audio or relied on laborious open-ended evaluation.

(3) The authors also contribute Proactive-Instruct-8K for training and train the Qwen3-Omni-Proactive model, which can be useful for the community.

Critical weaknesses

(1) While introducing PIFBench is a primary contribution of this paper, Section 3.3 significantly oversimplifies the data construction process. Crucial methodological details such as negative sample generation, response format variants, and human verification standards are entirely relegated to the appendices. This excessive reliance on the appendix undermines the standalone completeness of the main text.

(2) Both the Proactive-Instruct-8K and PIFBench heavily rely on multimodal models for automatic annotation and quality filtering, which might introduce noisy or imperfect supervision signals.

(3) Given the focus on streaming video and real-time proactive interaction, the paper lacks a quantitative analysis of memory usage, inference latency, or throughput.

(4) The use of a <Silence> token is an incremental tweak that mimics existing paradigms, such as EOS prediction in VideoLLM-Online. More critically, forcing the model to autoregressively generate CoT reasoning at every temporal chunk to balance training labels introduces severe inference latency. This design trades essential real-time throughput for label balance, fundamentally contradicting the latency-sensitive requirements of streaming applications and rendering the approach highly impractical for real-world deployment.

(5) Lack of discussion of recent papers on streaming proactive video understanding [1] - [3]

[1] StreamReady: Learning What to Answer and When in Long Streaming Videos, 2026 CVPR

[2] StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistant, 2025 NIPS

[3] Streaming Video Instruction Tuning, 2026 CVPR

Questions for authors

Please see the weakness.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 77ta

Dear Reviewer 77ta,

Thank you for your careful reading of our paper and for your constructive feedback. We respond to your comments point by point below.

W1: We thank the reviewer for raising this point. In the initial submission, we aimed to make the core construction process self-contained by describing the data sources, task taxonomy, and task-specific pipelines in Section 3.3 and summarizing them in Figure 2. To meet the page limit, we placed supplementary procedures for negative samples, response format variants, and benchmark conversion in the appendix. These components increase the benchmark’s diversity. Section 3.3 describes the essential steps of data construction. With the additional page in the revised version, we have added descriptions of negative sample generation, response format variants, and benchmark conversion to Section 3.3, while retaining fine-grained procedural details in the appendix for conciseness.

W2: We thank the reviewer for raising this concern. For PIFBench, multimodal models generate candidate annotations, which are then reviewed, corrected, or removed by human annotators according to explicit criteria. For Proactive-Instruct-8K, we agree that automatic annotation and filtering may leave some imperfect supervision. We use this approach because training data must be scalable, and verifying every sample manually is impractical. Our fine-grained pipeline decomposes annotation into constrained steps that are more manageable for multimodal models, helping us balance scalability and data quality. The improvement over the base model suggests that Proactive-Instruct-8K provides useful supervision, although it does not imply that all annotations are error free. We will release the dataset to support community inspection and refinement, and will use feedback to inform future improvements.

W3: Our method does not modify the model architecture or add inference modules, it only adds a special token to indicate when the model should remain silent. Thus, the memory required to store the model parameters remains unchanged. Since the base model does not support proactive interaction, directly comparing its efficiency with the adapted model would conflate the cost of enabling proactive interaction with the cost of our training strategy. We therefore added a controlled comparison between the CoT and non-CoT variants in Section 5.3.3. Under the same evaluation setting, we report their overall inference runtime and response length to characterize the efficiency tradeoff introduced by CoT. We thank the reviewer for encouraging us to include this analysis.

W4: We do not present the <Silence> token as a novel contribution in Section 4.2. Our main goal is to use PIFBench to evaluate proactive interaction and, since this capability is not explicitly supported by general-purpose models, to improve it through training data and establish a baseline. The token is a simple mechanism for expressing the silence decision. We agree that generating CoT at every chunk trades inference efficiency for accuracy. As noted in Section 5.3.3, we recommend the CoT-augmented variant because it provides richer supervision and achieves better overall performance. However, explicit CoT need not be retained in every deployment setting. Users can shorten or omit it when lower latency is more important, while accepting a possible reduction in accuracy. We have revised the wording in Section 5.3.3 to clarify this tradeoff and added: 'For applications that prioritize lower runtime, the non-CoT variant may be preferable.' We thank the reviewer for raising this important concern.

W5: We were aware of these works when conducting our study, but the initial related-work discussion was selective due to space constraints. In the revised version, with the additional page, we have added discussions of StreamReady, StreamBridge, and Streaming Video Instruction Tuning. We describe the first two as method-focused approaches and the third as work on streaming video instruction data, and clarify their relationship to our work. We thank the reviewer for drawing our attention to these relevant papers and helping us improve the discussion.

We hope that our revisions and clarifications address your concerns. Thank you again for your thoughtful feedback, which helped us improve the paper.

Sincerely,

The Authors

AuthorsOpenReview ↗

Looking forward to Feedback as Discussion Deadline Approaches

Hi, we sincerely thank you very much for these constructive comments and evaluation of our manuscript. As the discussion phase will be closed on November 18 11:59pm AoE, we would like to kindly ask you to take a look at our responses and reevaluate our work based on our clarifications. Please let us know whether our response addresses your concerns or whether there is any further detail we can provide to help address these concerns.

Thank you again for dedicating your time to reviewing our paper.

Thank you for the revisions. W1 and W5 are resolved, and I appreciate the new CoT comparison. Two concerns remain. The annotation quality of Proactive-Instruct-8K is still unquantified, and the aggregate runtime in Table 6 leaves per-chunk latency in streaming use unclear. I therefore maintain my rating of 2.

AuthorsOpenReview ↗

Response to Revierwer 77ta

Dear Reviewer 77ta,

Thank you for your feedback. We’re glad our earlier response addressed some of your concerns. We’d like to clarify the two remaining points.

1. Annotation quality of Proactive-Instruct-8K

To measure annotation quality, we built a simple agent loop to check examples for errors and hallucinations and to try to fix them. The prompt is presented in Appendix J. Given the time and cost limits, we have checked about 300 examples so far. We used a 0–10 scoring scale and obtained the following results.

ScoreMeaningShare
0–3Contains severe hallucinations7.7%
4–6The instruction does not match the task well19.7%
7–9Usable and high quality70.7%
10Meets all quality criteria2.0%

After a simple correction pass, 47.8% of the examples that initially scored 0–3 were rescored in the 7–9 range. This initial check suggests that most of the examples we reviewed are usable and of good quality. As shown in our paper, we have observed higher evaluation scores after training on Proactive-Instruct-8K, which supports its usefulness as training data.

Manually reviewing and correcting every example can be costly for us. Our pipeline makes it possible to generate this type of data at scale and at low cost. We will release the dataset publicly so other researchers can use it and build on it.

2. Per-chunk latency in streaming

Latency depends on the GPU hardware and the inference engine, so we compare all models under the same GPU setup. Different examples can produce different chunks, and the chunks can vary even when the same model processes the same example more than once. We therefore report the average end-to-end runtime across examples, including model loading and other overhead. Averaging over multiple examples helps reduce the effect of fluctuations in individual runs.

We know the discussion deadline is approaching, but we’d still like to do our best to address any remaining questions or concerns. Your feedback helps us make our work more solid. Thank you again, and we hope you’ll have a chance to read our response and share your thoughts before the discussion closes.

Reviewer_eUTFOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents PIFBench, a benchmark with 1,993 examples across 13 tasks that evaluates proactive instruction following. The benchmark uses rule-based evaluation to assess response decisions, content, and timing. The authors also construct Proactive-Instruct-8K and fine-tune Qwen3-Omni with it. The resulting model shows certain level of improvements.

Critical strengths

The work is substantial, covering task definition, dataset curation, and model training. Rule based evaluation is scalable and verifiable for RL systems

Critical weaknesses
  1. Including a representative example for each task category in the appendix would make the tasks easier to understand.
  2. How does the model perform under offline evaluation after training with the online streaming format?
  3. ICLR 2027 requires an AI use statement, but the paper places its disclosure at the end of the appendix under "Use of Large Language Models." This may pose a risk of being desk reject.
  4. In-depth analysis is insufficient. The paper does not thoroughly investigate how to address the silence-response imbalance, and does reinforcement learning further help performance gain?
  5. Metrics for evaluating performance under label imbalance, such asF1, False Positive Rate and False Negative Rate , are missing. The authors could select appropriate metrics to report, and analyze.
Questions for authors

Please address the weakness mentioned above

2 comments
AuthorsOpenReview ↗

Response to Reviewer eUTF

Dear Reviewer eUTF,

Thank you for your careful reading of our paper and for your constructive feedback. We respond to your comments point by point below.

W1: We thank the reviewer for this helpful suggestion. We have added representative examples for each task category in Appendix C (Table 8) to make the tasks easier to understand.

W2: We thank the reviewer for this question. We additionally evaluated Qwen3-Omni-Proactive under the offline protocol. Its TotalAcc, ReactAcc, RespAcc, and WT-MAE are 12.25, 78.24, 43.56, and 33.82, respectively, compared with 11.31, 81.41, 40.84, and 33.70 for the original Qwen3-Omni model. The results show only modest differences. We hypothesize that this may be due to our use of LoRA rather than full-parameter fine-tuning and the mismatch between the online training format and offline evaluation. We report these results and discuss them in the appendix to avoid conflating the different evaluation protocols in Table 3.

W3: We thank the reviewer for pointing this out. We have revised the disclosure section’s title and content and moved it to after the conclusion and before the references, in accordance with the ICLR 2027 guidelines.

W4: We believe that we have investigated the silence-response imbalance, as described in Section 4.2. To restate our rationale, our goal was to provide a strong baseline while preserving generalizability. We therefore avoided modifying the model architecture and instead added a special <Silence> token to indicate when the model should remain silent. We soon observed the silence-response imbalance. As shown in Table 5, the PIF data w/o CoT setting has the lowest ReactAcc among our trained variants. In pilot experiments, we tried reweighting the silence and reaction tokens in the loss function and applying focal loss to address this imbalance. These approaches did not yield satisfactory overall improvements. We therefore introduced CoT to balance the supervision and selected this setting for our final baseline. Although our experiments focus on SFT, we provide detailed failure-mode, subtask analysis and case analysis in Appendix F due to space constraints. We have not explored reinforcement learning in this work, but consider it a promising direction for our future research. Thanks again for your suggestion.

W5: We thank the reviewer for this suggestion. In PIFBench, positive and negative samples indicate whether an instruction requires a response, but PIF is not simply a binary classification task. Positive samples may contain multiple required responses with specific content and timing, whereas negative samples have no target response content or timestamp. Therefore, F1, FPR, and FNR based only on binary response decisions would not capture whether the model provides the correct content at the correct time. We use TotalAcc as the unified metric because it evaluates whether the model responds correctly and at the right time when required, and remains silent when no response is needed. ReactAcc, RespAcc, and WT-MAE provide complementary diagnostic views of these aspects.

We hope our clarifications and additional analyses address your concerns. We thank you again for your constructive feedback, which helped us improve the paper.

Sincerely,

The Authors

Reviewer_eUTFOpenReview ↗

Reviewer Response

Thank you for the clarifications and additional analyses. They address most of my concerns. I have updated my scores. Good luck to authors.

Reviewer_uKwUOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies proactive interaction in streaming video understanding. The authors formalize Proactive Instruction Following (PIF): an instruction is given before a video starts, and the model must process the stream incrementally, remain silent until the specified condition is met, and then produce the required response within a valid time window. They build PIFBench (1,993 samples, 13 tasks organized by modality × temporal dependency) with rule-based verification. They further construct Proactive-Instruct-8K and adapt Qwen3-Omni-30B-A3B-Instruct into Qwen3-Omni-Proactive via a <Silence> token, timestamped streaming training format, and CoT-augmented supervision that mitigates the extreme silence–reaction label imbalance

Critical strengths
  1. The task formalization makes open-ended proactive interaction programmatically verifiable. By design of "pre-given instruction + exact response content + valid time window," PIF converts open-ended interactive behavior—which previously required costly human judgment—into a rule-verifiable instruction-following problem, enabling large-scale and reproducible evaluation. 2: The multi-dimensional experimental design provides a reliable basis for capability analysis. The paper systematically evaluates three representative model groups (omni-modal models without proactive capability, visual-only models with proactive capability, and omni-modal models with proactive capability), and two controlled comparisons—offline vs. online protocols and original vs. This layered attribution analysis strengthens the credibility of the paper's conclusions.
Critical weaknesses

The real-time cost of CoT inference is not evaluated. Table 2 suggests the model must generate a CoT trace at every streaming step, including the many steps that end in silence. This overhead may weaken real-time responsiveness, which is critical for streaming video. However, the paper reports no latency or CoT length, and timing is measured against chunk timestamps, not wall-clock time. Given the small gain over the non-CoT variant, the authors should report per-step latency relative to the chunk interval.

Questions for authors
  1. ReactAcc on positive samples only counts how many reactions are emitted. As a result, a model that keeps responding can reach about 90% ReactAcc. Is the proportion of negative samples (~10%) too low to penalize such over-triggering? A false-trigger rate, or results reported separately for positive and negative samples, would help.
  2. The CoT and non-CoT variants differ only slightly in TotalAcc (19.42 vs. 18.73), and the non-CoT variant actually achieves higher RespAcc (65.09 vs. 57.99). Could the authors explain this result and report variance across multiple runs to show whether the gap is significant?
3 comments
AuthorsOpenReview ↗

Response to Reviewer uKwU

Dear Reviewer uKwU,

Thank you for your careful reading of our paper and for your constructive feedback. We respond to your comments point by point below.

W1: We thank the reviewer for this valuable suggestion. As the reviewer notes, 'timing is measured against chunk timestamps, not wall-clock time.' We use chunk-level evaluation because evaluating at 1-second intervals would require substantially more repeated inference, increasing evaluation cost and the risk of out-of-memory errors. Since the number of response turns per sample differs between the CoT and non-CoT variants, per-step runtime would not provide a directly comparable measure. We therefore added an analysis in Section 5.3.3 comparing average runtime and response length per sample, which better reflects the average generation cost across complete samples. We report the results below:

SettingAverage response length (tokens/sample)Average runtime (GPU-seconds/sample)
PIF data w/o CoT11485.6
PIF data w/ CoT2,510248.5

Runtime is based on the average script execution time on 32 A100 GPUs. We have also revised the discussion in Section 5.3.3 to clarify this evaluation choice. We thank the reviewer for encouraging us to report these efficiency measurements.

Q1: We thank the reviewer for raising this concern. ReactAcc, RespAcc, and WT-MAE are diagnostic metrics that complement TotalAcc. A model that responds frequently may achieve a high ReactAcc on positive samples, but this score alone does not indicate strong overall performance. For example, in our non-CoT + focal loss ablation, ReactAcc is 73.71, while TotalAcc is 10.55 and WT-MAE is 107.42. This contrast shows that ReactAcc should be interpreted alongside the other metrics. We set the proportion of negative samples to approximately 10% as a deliberate balance. PIFBench primarily evaluates whether a model produces the required response content at the appropriate time when a trigger occurs. Negative samples have no valid response content or response time, so increasing their proportion would reduce coverage of these core evaluation targets. We retain enough negative samples to assess whether models remain silent when no trigger occurs. This choice also reflects our focus on instructions targeting events that may occur in the video stream, rather than conditions that are guaranteed never to occur. We interpret 'false-triggers' as responses emitted when the ground truth requires the model to remain silent, including responses on negative samples and responses outside the valid response windows in positive samples. ReactAcc and WT-MAE provide complementary information: ReactAcc evaluates reaction decisions, while WT-MAE quantifies the temporal deviation of correct-content responses on positive samples. In this respect, WT-MAE provides information about the severity of timing errors beyond a binary false-trigger rate.

Q2: We thank the reviewer for raising this question. We ran three experiments for each variant and report the mean and standard deviation below:

SettingTotalAccReactAccRespAccWT-MAE
Non-CoT17.74 ± 0.9449.82 ± 2.9065.98 ± 2.5462.08 ± 3.65
CoT19.31 ± 0.6165.63 ± 2.6958.68 ± 1.5556.14 ± 3.46

CoT achieves higher TotalAcc. A closer look shows that it also achieves higher ReactAcc, indicating that the model is less likely to remain silent when a response is required. Its RespAcc is lower than that of non-CoT, which may indicate that the model does not learn the CoT supervision well enough to judge response content accurately. We introduced CoT primarily to address the low ReactAcc of the non-CoT variant. Because silence tokens dominate the training data, the model tends to over-predict silence. CoT is intended to mitigate this behavior. The lower ReactAcc of non-CoT suggests that the model becomes less confident and less willing to respond. Without CoT, there is also no explicit reasoning trace to inspect when the model remains silent or to understand its decision path. We therefore believe CoT offers benefits beyond the score improvement: its traces may support root-cause analysis and future training-set refinement. We consider these directions for future work. Since not all training ablations were repeated across three runs, we leave the original results in Table 5 unchanged and report the three-run CoT and non-CoT results in the appendix.

We hope our clarifications and additional analyses address your concerns. We thank you again for your constructive feedback, which helped us improve the paper.

Sincerely,

The Authors

AuthorsOpenReview ↗

Looking forward to Feedback as Discussion Deadline Approaches

Hi, we sincerely thank you very much for these constructive comments and evaluation of our manuscript. As the discussion phase will be closed on November 18 11:59pm AoE, we would like to kindly ask you to take a look at our responses and reevaluate our work based on our clarifications. Please let us know whether our response addresses your concerns or whether there is any further detail we can provide to help address these concerns.

Thank you again for dedicating your time to reviewing our paper.

Dear Reviewer uKwU,

As the discussion deadline approaches, we’d like to follow up on our earlier response. If you have any remaining questions or concerns, we’d be happy to clarify.

Thank you again for your time and feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.