acceptodds
Under review as a conference paper at ICLR 2027

Before You Ask: Learning to Proactively Initiate Interaction in Streaming Video

Abstract

Streaming video assistants require model to provide timely, context-aware support while remaining silent when intervention is unnecessary. Existing methods remain fundamentally reactive, relying on explicit user queries to trigger responses, thus failing to autonomously initiate assistance in dynamic real-world scenarios. To bridge this gap, we propose a zero-trigger paradigm that shifts the objective from merely deciding when to respond to autonomously determining whether, when, and how to intervene. Specifically, we replace the conventional binary speak/silence actions with four functional tokens: inquiry, response, notify, and silence. To realize this, we introduce SAIWen, a streaming multimodal model post-trained via a two-stage framework on SAIWen-Interact, a dataset of 26,549 training and 2,000 test videos with temporally aligned dialogues. During Supervised Fine-Tuning (SFT), our a Hierarchical Class-Balanced Loss decouples macro-level intervention triggering from micro-level functional routing, mitigating extreme functional token imbalance without degenerating into trivial always-silent policies. Subsequently, Reinforcement Learning (RL) with a Curriculum Advantage Annealing directly optimizes temporal alignment against our MII-Score, which rigorously evaluates proactive inquiry quality and notification precision while penalizing passive abstention. Experiments demonstrate that SAIWen significantly advances autonomous video assistance, improving intervention precision by 30.32% and notification timeliness by 14.90%, while strictly minimizing interruption costs.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Streaming Video Understanding, Proactivate Interaction

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_1HT4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a model-initiated interaction task for streaming multimodal assistants, where the model decides on its own whether, when, and how to talk to the user, without an initial user query. To do this, the paper replaces the binary speak/silence decision with four action types: inquiry, response, notify, and silence.

The paper's contributions include: (1) SAIWen-Interact, a machine-generated dataset of temporally aligned video dialogues (26.5K train / 2K test videos), (2) MII-Score, a metric combining an LLM-judged inquiry score and a notification score that penalizes late or redundant notifications, and (3) SAIWen, a Qwen3.5-4B model trained with SFT using a Hierarchical Class-Balanced Loss, followed by RL with Curriculum Advantage Annealing.

SAIWen outperforms the baselines on SAIWen-Interact and improves over its base model on existing streaming video benchmarks.

Critical strengths
  1. The proposed task is interesting and novel. Most proactive streaming models still require an initial user request before they start monitoring. This paper instead lets the model decide whether, when, and how to start an interaction, and splits "speak" into inquiry, notify, and response. This is a meaningful step toward ambient assistants, and it is the main reason the paper is relevant to the community.

  2. The dataset and its construction approach are potentially useful. As the paper notes, no public dataset currently targets model-initiated streaming interaction. SAIWen-Interact's pipeline, which grounds the opening inquiry in an estimated activity-confirmation time and aligns notifications to key events, is a reasonable way to build such data at scale.

Critical weaknesses
  1. The motivation for model-initiated interaction is unclear.

In practice, interaction should be bidirectional: either the user or the model may start the conversation. Letting the model initiate can be a useful capability, but the paper gives no reason why the model should always initiate first, and some users may simply want answers to what they ask. This is arguably an open question that touches on UX/HCI, but since model-initiated interaction is the paper's central premise, it needs to be motivated.

This matters all the more because the framework may come at a cost to the conventional user-initiated setting. On the query-based streaming benchmarks (Table 3), SAIWen improves over its Qwen3.5-4B base, but similarly sized general models report higher scores. For example, Qwen3-VL-4B reports 54.5 / 57.4 / 80.2 on OVBench / ODVBench / StreamingBench-RT, compared with SAIWen’s 48.2 / 53.3 / 76.9. The paper does not analyze whether training for model-initiated interaction trades off user-initiated performance. If it does, a clear justification for always initiating first becomes essential.

Relatedly, prior work is mischaracterized. The paper describes existing proactive approaches as "fundamentally reactive" (Abstract; Sec. 1; Sec. 3.1). I agree that most of them assume an initial query, but once instructed they decide autonomously when and what to say, so calling them reactive overstates the gap.

  1. The dataset, metric, and method are not sufficiently validated.

(a) Dataset.

  • SAIWen-Interact is entirely machine-generated. The validation step in Sec. 3.2 (step 5) only checks temporal accuracy and role validity. Nothing checks whether the generated events, activity-confirmation times, or dialogues are actually correct.
  • There is no human verification during dataset construction, and no estimate of label quality.
  • Table 1 is used to argue that windowing alleviates the class imbalance. However, the chosen configuration (17.72) is not the lowest ratio: event-dense clips give 15.78, and event-focused + event-dense give 15.90. It is unclear why 17.72 should be considered better than these or than the full-video baseline (20.51).

(b) Metric.

  • The inquiry score depends on a judge model. That use differs from the conventional text-only comparison, because the judge must decide whether an inquiry fits the ongoing activity.
  • The paper neither names the judge model nor reports its agreement with human ratings, so the reliability of Q_inquiry, which makes up half of the MII-Score, is unknown. The notification quality q used in U_notify is described as a "semantic matching quality," without specifying how it is computed.
  • This matters because most of the headline gain comes from this component. Against JoyAI-VL-Interaction (Table 2), Q_inquiry rises by +28.7 (28.00 → 56.68) while U_notify rises by +14.4 (1.65 → 16.05), so about two-thirds of the MII-Score improvement (14.83 → 36.37) rests on the least-validated measurement: a judge rating of a single opening sentence.
  • MII-Score evaluates only <inquiry> and <notify>. Response quality, appropriate silence, and the timing of the inquiry are never measured; false notifications in silent regions are logged but not reported (Sec. 4.2). Two of the four proposed actions are therefore unevaluated, which also leaves the need for the four-way taxonomy unsupported (see Question 1).

(c) Method.

  • The ablations show mixed results. In Table 4, the default loss-weight ratio gives the best MII-Score but not the best EPAUC or P_notify. In Table 6, fixed-weight GRPO lowers the inquiry score (55.30 → 51.25), and the full RL pipeline improves MII by only +1.63 over SFT (30.05 → 31.68), on a 100-video subset with no variance reported.
  • Table 3 lacks relevant streaming baselines such as VideoChat3 and MMDuet2.
  • Comparing against the Qwen3.5-4B base model alone is not sufficient. Most of the improvement may come from SFT on in-domain data, especially since the RL ablation shows only a small and mixed effect.

Since all main claims rest on SAIWen-Interact and the MII-Score, these validation gaps make it hard to assess how much of the reported improvement reflects real capability.

Questions for authors
  1. Why four types? Could the action space be reduced to <response> vs. <silence>, and could the model be trained with only two labels? That seems a more generalizable and scalable approach. An ablation would clarify whether the finer taxonomy is necessary.
  2. Was any part of SAIWen-Interact verified by humans? Please report label accuracy or agreement on a sample.
  3. Which model is the judge for Q_inquiry, and how well does it agree with human ratings?
  4. Table 1: why is a silence:speak ratio of 17.72 preferable to 15.78, 15.90, or 20.51? What is the target ratio, and how was it chosen?
  5. How does an SFT-only model trained on the same in-domain data compare with the full pipeline on the full test set? How much variance is there across runs?
  6. Does training on SAIWen-Interact affect user-initiated performance? Comparing the same backbone fine-tuned with and without SAIWen-Interact on the Table 3 benchmarks would clarify whether model-initiated training comes at a cost.
2 comments
AuthorsOpenReview ↗

Response to Reviewer 1HT4

Thank you for your constructive reviews and we will address your concerns as follows.

W1: The motivation for model-initiated interaction.

R: We apologize for the misunderstanding and agree with the reviewer; we wish to clarify that we do not expect the model to always initiate first. Our reasoning was that, because the training data are predominantly query–assistant style, we initially suspected that the model might struggle to learn the model-initiate paradigm. This paper therefore focuses primarily on whether the model can learn to proactively initiate inquiries when the user issues no query. We believe that a model capable of relying on streaming video rather than on user queries is highly meaningful. Regarding the metrics, we could not find official results on OVOBench and other datasets. We attempted to investigate this, but the metrics reported by OneStreamer also differ from those of MaLoW. This is mainly because OVOBench itself defines both strict and lenient protocols. In this paper, we consistently adopt a single configuration, which ensures the validity of our evaluation. As for the passive formulation, we will revise this description in the main text and adopt the established term proactive.

Q1: The choice of four types.

R: In our thought, binary and four types address different levels of questions. A Response/Silence binary decides whether to produce language at a given moment, but does not distinguish proactive intent confirmation, responses to received user input, and event notifications when no new question arises, behaviors with different triggers, interaction states, and supervision targets. For example, the same utterance ("Do you need any help?") may be a proactive inquiry or a response to the previous turn; a speak/silence label cannot express this functional difference. The four action types make these states explicit: Inquiry establishes helping intent, Response handles user input, Notify provides autonomous event assistance, and Silence preserves the option to keep observing. This refinement clarifies the types of model responses and better supports proactive questioning.

Q2: The human verification of human evaluation.

R: Our main focus is on designing a data pipeline akin to Flora. In our construction procedure, we first validate the effectiveness of the method at a small scale before proceeding to large-scale data construction. In future work, we will attempt to manually verify the entire dataset.

Q3: The clarification of windowing alleviates.

R: We wish to clarify this point. In Table 1, the Event-centric views are primarily intended to show the ratio of silence tokens to speak tokens in the windows obtained after cropping the video using the two Temporal windowing schemes. The full + event-centric views report the overall ratio after incorporating temporal windowing. Our aim is to show that data constructed with temporal windowing can substantially mitigate the imbalance in the silence-to-speak token ratio.

Q4: The human verification of the score and the choice of llm.

R: We adopt an LLM-as-a-judge approach primarily to accommodate the online training setting. Because it is difficult to predetermine when the model should ask and what it should ask, we need an LLM to serve as an immediate responder within the program rather than relying on manually specified replies. In Appendix A.5, we report manual evaluation results showing that LLM-as-a-judge achieved acceptable scores on most videos. We believe that a better-trained model could be used for this task in future work. As for the judge model, we selected Qwen3.8-27B and will state this in the main text (line 234). Semantic matching quality is determined by the LLM Judge, which assesses the semantic consistency between a model notification and the reference event, assigning scores of 0, 1, and 2 for mismatch, partial match, and sufficient match, respectively, which are then normalized to [0, 1].

Q5: The clarification of MII-Score on silence.

R: Actually, MII-Score does not ignore silence; rather, it implicitly evaluates silence decisions through event coverage and notification precision. Unnecessary Notify actions increase the precision denominator without yielding additional valid matching gains, whereas remaining silent when a reminder is warranted incurs a loss in event coverage. The metric therefore encourages speaking when appropriate and remaining silent when not, rather than separately rewarding the number of silent decisions, thereby avoiding the degenerate strategy of remaining silent throughout.

Thank you for the response. Some concerns are addressed, but some are not addressed. The motivation is still unclear, and I think the choice of four action types should be supported by experiments comparing the different action types. The data validation remains unclear, and some questions (Q5, Q6) are not covered.

Hence, I will keep my score.

Reviewer_Mi69OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies model-initiated streaming-video interaction without an initial task request. SAIWen recognizes an activity, asks whether assistance is wanted, and subsequently responds, notifies, or remains silent. The authors introduce SAIWen-Interact with 26,549 training and 2,000 test videos, four functional decision tokens, a class-weighted focal SFT objective, a phased RL advantage schedule, and MII-Score. The intended contribution is the capability to identify assistance opportunities and confirm user intent. The reported experiments establish initiation in selected actionable activities more directly than reliable decisions about whether intervention is appropriate in an unrestricted stream.

Critical strengths
  1. A worthwhile interaction objective. Explicit intent confirmation addresses a meaningful issue beyond answering a user-specified event: an assistant should establish whether help is wanted before providing it. This could support an important capability contribution if appropriate restraint is demonstrated.
  2. Useful structured data and initial evaluation machinery. The dataset provides temporally aligned dialogues and functional actions at a substantial scale. Combining timeliness-sensitive notification quality with one-to-one precision is also more informative than checking response content alone.
Critical weaknesses
  1. The evaluation does not establish appropriate initiation and restraint, which are central to the contribution. Section 3.2 selects interaction-suitable activities, without a systematic evaluation of complete no-help, ambiguous-intent, and unwanted-assistance streams. Inquiry scoring explicitly ignores timing, and notification precision does not directly measure all inquiry/response interruptions or refusal violations. Under Equation (1), absent an additional eligibility rule, a valid perfect inquiry followed by no notifications receives 50/100, above the reported model score of 36.37. This is a metric counterexample, not an observed baseline result. Figure 10 further labels a period as following refusal but shows five notifications at 5–19 seconds; its note says the source still contains notifications and the silence panel is schematic. The source must be clarified, and this figure cannot establish reliable return to silence. These issues undermine the claimed capability rather than merely its presentation.
  2. The reported gains do not isolate the proposed learning contribution. Appendix A.4 gives SAIWen up to 200 rolling frames, whereas prompt-driven baselines receive the first frame plus four recent frames and eight dialogue turns. Matching decision times does not match available evidence. The native base-model comparison is useful, but does not separate new data from the proposed objectives. Equation (7) gives all active classes the same per-token weight, making the claimed hierarchy principally active/silence reweighting plus focal modulation. Table 6's curriculum gain is only 1.63 MII points over SFT and 1.14 over fixed-weight GRPO, without uncertainty; its EPAUC is lower than fixed-weight GRPO. These confounds leave insufficient evidence that the particular objectives, rather than information access or ordinary task training, explain the advantage.
Questions for authors
  1. Metric clarification: What additional rule, if any, prevents an inquiry-only policy from receiving 50/100? How are unsupported inquiry timing, repeated inquiries, and continued speech after refusal reflected in the reported score?
  2. Capability evidence: What are the false-inquiry and refusal-violation rates on no-help, ambiguous-intent, and rejection cases? Is Figure 10 a model rollout or a data example, and how should its post-refusal notifications be interpreted?
  3. Decisive controls: Do the improvements persist with matched observation histories and identical-data comparisons against ordinary weighted/focal SFT and tuned fixed-weight RL? Are the small curriculum gains statistically stable?
  4. Evaluation clarification: Were default tolerances selected using the 100-video test subset, as Section 5.2 appears to indicate? Please identify an independent validation protocol and the exact comparators/aggregation behind the abstract's 30.32% and 14.90% improvements.
1 comment
AuthorsOpenReview ↗

Response to Reviewer Mi69

Thank you for your constructive reviews and we will address your concerns as follows.

W1: The clarification of our scope.

R: We thank the reviewer for this suggestion. Complete no-help, ambiguous-intent, and unwanted-assistance streams are indeed important issues in interaction. However, the issues mentioned above constitute a very broad problem, and fully resolving them within a single study is clearly unrealistic. Recent work has gradually decomposed interaction into interruption and ambiguous questioning, and we also made corresponding attempts in our work, although these did not yield significant improvements. Based on these considerations, this work mainly investigates whether the model can proactively discover events and provide assistance when the user issues no query. We will explore these directions in the future.

W2: The clarification of the detailed learning contribution.

R: We wish to clarify several details. Regarding window settings, we keep them consistent with the model defaults so that inference aligns with training; prior work has also typically used the maximum number of frames permitted by model capacity. Meanwhile, Appendix A.4 explicitly discloses the context settings of different inference mechanisms: prompted systems re-encode short windows at each step, whereas native streaming models continuously maintain history. The same decision grid provides equal speaking opportunities, but we do not interpret this as identical visual evidence. To reduce interface differences, we provide a comparison using the Qwen3.5 backbone with the native path. As for the reinforcement learning results, because proactively initiating speech and responding to user queries are opposing paradigms, the final outcome is essentially a trade-off.

Q1: The clarification of score and error of Figure 10.

R: There may be a misunderstanding on this point: a model does not automatically receive a score of 50/100 simply by issuing a query at the outset. In Appendix A, we provide the detailed scoring prompt. The model is expected to accurately infer the user's likely intent from the available streaming information and, based on this inference, proactively provide assistance. Whether the query correctly identifies the target task and the target entity therefore affects the query's score. We apologize for the error in Figure 10. When the user explicitly declines, the model remains silent rather than responding; we have corrected this. We thank the reviewer for pointing out this issue.

Q2: The control of the experiments.

R: Although this question is somewhat difficult to parse, we understand it as asking whether weighted/focal SFT and tuned fixed-weight RL remain effective when the historical outputs are consistent with those in the samples. Our answer is yes: in practice, the model's actual outputs often diverge from those in the samples, and if a matching streaming output can be provided, the model will operate in a cleaner environment and thus produce better outputs. Weighted/focal SFT, by contrast, primarily aims to mitigate the imbalance caused by an excessive proportion of silence in the training samples and is not affected by historical outputs.

Q3: The clarification of the sampled cases.

R: For evaluation, we randomly sampled 100 examples from the original dataset; this random sampling ensures that our testing is fair. We also used a low temperature setting in our experiments, which reduces the stochasticity introduced by model inference.

Reviewer_2ymUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a “model-initiated interaction” paradigm in which a streaming video model autonomously decides whether, when, and how to intervene, rather than waiting for an explicit user query. The authors introduce four functional tokens (inquiry, response, notify, silence), a synthetic dataset (SAIWen-Interact), a new evaluation metric (MII-Score), and a two-stage training framework (SFT with Hierarchical Class-Balanced Loss and RL with Curriculum Advantage Annealing).

Critical strengths
  1. Clear task formalization:Extending the binary speak/silence action space to four functional tokens, and decoupling “whether to intervene” from “how to intervene,” is a reasonable task redesign.

  2. Purpose-built dataset and evaluation metric: The introduction of the SAIWen-Interact dataset alongside the MII-Score would help advance the development of proactive response tasks.

Critical weaknesses
  1. Unverified Evaluation Metric (MII-Score): The MII-Score relies entirely on an LLM-as-a-judge. The paper fails to provide a rigorous, large-scale human evaluation to prove that MII-Score aligns with actual human preference. Also, relying on black-box LLM judges for RL optimization and benchmarking at the same time is known to be susceptible to reward hacking.

  2. Confounded "No Degradation" Claim (Table 3): §5.1 asserts that proactive training "enhances temporal reasoning" based on SAIWen's gains over the Qwen3.5-4B base in Table 3. However, this comparison does not isolate the proposed mechanism: SAIWen's SFT mixture already incorporates auxiliary StreamO streaming instances and offline QA samples, which alone could account for the gains over the base model. The cross-model evidence is also mixed rather than uniformly favorable: at the same 4B scale, SAIWen trails VideoChat-Online 4B on OVBench (48.23 vs. 55.20) and ties on ODVBench (53.28 vs. 53.65), despite leading on OVOBench and StreamingBench. The "enhances temporal reasoning" claim is therefore not supported by the reported evidence.

Questions for authors
  1. Could you fine-tune at least one streaming baseline (e.g., JoyAI-VL-Interaction) on SAIWen-Interact and re-run the comparison? This would isolate the contribution of your training framework from the data advantage.
1 comment
AuthorsOpenReview ↗

Response to Reviewer 2ymU

Thank you for your constructive reviews and we will address your concerns as follows.

W1: The verification of evaluation metric (MII-Score).

R: We understand that the reviewer may have concerns regarding LLM-based scoring; in fact, we manually evaluated a subset of samples in Appendix A.5 and also conducted sampling-based checks of the assigned scores. To illustrate MII-Score computation, we selected a 45-s egg-ingredient video and verified each stage:

Stage / Ref. timeModel output and manual verificationJudge score
Initial inquiry / 2 sAt 1 s, proactively asked whether to track egg mixing; contextually appropriate but broad.3/4
Extra notifications / 3–8.5 sThree notifications matched no reference event; all counted in the precision denominator.No match
Salt / 12 sNotifications at 12.5 s and 18 s failed to describe salt addition.0/2, 0/2
Chili / 16 sNotifications at 18, 22.5, and 23 s did not cover this event.0/2 each
Baking powder / 24 sFirst two notifications described only preparation; the 30-s notification correctly identified powder addition.0/2, 0/2, 2/2
Beating eggs / 33 s32.5-s notification correctly identified beating and received partial credit; 40-s notification received full credit.1/2, 2/2
Mixing / 41 s40-s notification accurately described the final mixture state.2/2

Using a tolerance window of 2 s before and 8 s after reference times, the five Notify events received coverage scores of 0, 0, 0.200, 0.475, and 0.833, yielding E = 0.301667. The 10 notifications had total one-to-one matching quality 2.5, so P = 0.25. With Q = 0.75 and U = 2EP/(E + P) = 0.273414, MII-Score = 100 × (Q + U)/2 = 51.17.

This case shows the full computation from semantic evaluation to final score: correctly identified events receive credit, while uncovered events and redundant notifications reduce the score; manual verification also revealed minor discrepancies in temporal annotations and semantic ratings.

W2: The comparison of SAIWen with other methods on traditional streaming task (Tbale 3).

R: We thank the reviewer for identifying these two distinct issues. The primary purpose of Table 3 is to examine whether, after introducing the new capability of model-initiated interaction, the complete training recipe still preserves video understanding performance in conventional user-query settings, rather than attributing all gains to any single data subset.

What's more, Being both 4B does not imply that the two models share the same pretrained model, training data, or training procedure. VideoChat-Online was trained on 1.6M samples, whereas SAIWen used only 45K; the performance of VideoChat-Online may therefore derive from its training data. This is not, however, the main point we wish to make. Rather, we aim to show that after training on Model-Initial data, the model can support both proactive interaction and traditional video understanding, that is, acquiring proactive capability does not necessarily come at the expense of traditional capability. We will add this clarification in a revision.

Q1: The finetune of JoyAI-VL-Interation.

R: Thanks for your advice, we are conducting the relevant experiments. Despite this, after discussion, we think that fine-tuning a streaming baseline would not yield conclusive results. From a theoretical standpoint, fine-tuning a conventional streaming baseline would involve fine-tuning four categories of decisions, temporal supervision, and the interaction protocol, followed by further fine-tuning; this would not automatically isolate the effect of the training framework more cleanly than a loss-controlled comparison using the same backbone and the same data. Moreover, fine-tuning the baseline would support only limited claims. We therefore prefer to use the conventional baseline to show that this task cannot be adequately covered by models trained on conventional data.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.