- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes a model-initiated interaction task for streaming multimodal assistants, where the model decides on its own whether, when, and how to talk to the user, without an initial user query. To do this, the paper replaces the binary speak/silence decision with four action types: inquiry, response, notify, and silence.
The paper's contributions include: (1) SAIWen-Interact, a machine-generated dataset of temporally aligned video dialogues (26.5K train / 2K test videos), (2) MII-Score, a metric combining an LLM-judged inquiry score and a notification score that penalizes late or redundant notifications, and (3) SAIWen, a Qwen3.5-4B model trained with SFT using a Hierarchical Class-Balanced Loss, followed by RL with Curriculum Advantage Annealing.
SAIWen outperforms the baselines on SAIWen-Interact and improves over its base model on existing streaming video benchmarks.
Critical strengths
-
The proposed task is interesting and novel. Most proactive streaming models still require an initial user request before they start monitoring. This paper instead lets the model decide whether, when, and how to start an interaction, and splits "speak" into inquiry, notify, and response. This is a meaningful step toward ambient assistants, and it is the main reason the paper is relevant to the community.
-
The dataset and its construction approach are potentially useful. As the paper notes, no public dataset currently targets model-initiated streaming interaction. SAIWen-Interact's pipeline, which grounds the opening inquiry in an estimated activity-confirmation time and aligns notifications to key events, is a reasonable way to build such data at scale.
Critical weaknesses
- The motivation for model-initiated interaction is unclear.
In practice, interaction should be bidirectional: either the user or the model may start the conversation. Letting the model initiate can be a useful capability, but the paper gives no reason why the model should always initiate first, and some users may simply want answers to what they ask. This is arguably an open question that touches on UX/HCI, but since model-initiated interaction is the paper's central premise, it needs to be motivated.
This matters all the more because the framework may come at a cost to the conventional user-initiated setting. On the query-based streaming benchmarks (Table 3), SAIWen improves over its Qwen3.5-4B base, but similarly sized general models report higher scores. For example, Qwen3-VL-4B reports 54.5 / 57.4 / 80.2 on OVBench / ODVBench / StreamingBench-RT, compared with SAIWen’s 48.2 / 53.3 / 76.9. The paper does not analyze whether training for model-initiated interaction trades off user-initiated performance. If it does, a clear justification for always initiating first becomes essential.
Relatedly, prior work is mischaracterized. The paper describes existing proactive approaches as "fundamentally reactive" (Abstract; Sec. 1; Sec. 3.1). I agree that most of them assume an initial query, but once instructed they decide autonomously when and what to say, so calling them reactive overstates the gap.
- The dataset, metric, and method are not sufficiently validated.
(a) Dataset.
- SAIWen-Interact is entirely machine-generated. The validation step in Sec. 3.2 (step 5) only checks temporal accuracy and role validity. Nothing checks whether the generated events, activity-confirmation times, or dialogues are actually correct.
- There is no human verification during dataset construction, and no estimate of label quality.
- Table 1 is used to argue that windowing alleviates the class imbalance. However, the chosen configuration (17.72) is not the lowest ratio: event-dense clips give 15.78, and event-focused + event-dense give 15.90. It is unclear why 17.72 should be considered better than these or than the full-video baseline (20.51).
(b) Metric.
- The inquiry score depends on a judge model. That use differs from the conventional text-only comparison, because the judge must decide whether an inquiry fits the ongoing activity.
- The paper neither names the judge model nor reports its agreement with human ratings, so the reliability of Q_inquiry, which makes up half of the MII-Score, is unknown. The notification quality q used in U_notify is described as a "semantic matching quality," without specifying how it is computed.
- This matters because most of the headline gain comes from this component. Against JoyAI-VL-Interaction (Table 2), Q_inquiry rises by +28.7 (28.00 → 56.68) while U_notify rises by +14.4 (1.65 → 16.05), so about two-thirds of the MII-Score improvement (14.83 → 36.37) rests on the least-validated measurement: a judge rating of a single opening sentence.
- MII-Score evaluates only
<inquiry>and<notify>. Response quality, appropriate silence, and the timing of the inquiry are never measured; false notifications in silent regions are logged but not reported (Sec. 4.2). Two of the four proposed actions are therefore unevaluated, which also leaves the need for the four-way taxonomy unsupported (see Question 1).
(c) Method.
- The ablations show mixed results. In Table 4, the default loss-weight ratio gives the best MII-Score but not the best EPAUC or P_notify. In Table 6, fixed-weight GRPO lowers the inquiry score (55.30 → 51.25), and the full RL pipeline improves MII by only +1.63 over SFT (30.05 → 31.68), on a 100-video subset with no variance reported.
- Table 3 lacks relevant streaming baselines such as VideoChat3 and MMDuet2.
- Comparing against the Qwen3.5-4B base model alone is not sufficient. Most of the improvement may come from SFT on in-domain data, especially since the RL ablation shows only a small and mixed effect.
Since all main claims rest on SAIWen-Interact and the MII-Score, these validation gaps make it hard to assess how much of the reported improvement reflects real capability.
Questions for authors
- Why four types? Could the action space be reduced to
<response>vs.<silence>, and could the model be trained with only two labels? That seems a more generalizable and scalable approach. An ablation would clarify whether the finer taxonomy is necessary. - Was any part of SAIWen-Interact verified by humans? Please report label accuracy or agreement on a sample.
- Which model is the judge for Q_inquiry, and how well does it agree with human ratings?
- Table 1: why is a silence:speak ratio of 17.72 preferable to 15.78, 15.90, or 20.51? What is the target ratio, and how was it chosen?
- How does an SFT-only model trained on the same in-domain data compare with the full pipeline on the full test set? How much variance is there across runs?
- Does training on SAIWen-Interact affect user-initiated performance? Comparing the same backbone fine-tuned with and without SAIWen-Interact on the Table 3 benchmarks would clarify whether model-initiated training comes at a cost.
2 comments
Thank you for the response. Some concerns are addressed, but some are not addressed. The motivation is still unclear, and I think the choice of four action types should be supported by experiments comparing the different action types. The data validation remains unclear, and some questions (Q5, Q6) are not covered.
Hence, I will keep my score.
Response to Reviewer 1HT4
Thank you for your constructive reviews and we will address your concerns as follows.
W1: The motivation for model-initiated interaction.
R: We apologize for the misunderstanding and agree with the reviewer; we wish to clarify that we do not expect the model to always initiate first. Our reasoning was that, because the training data are predominantly query–assistant style, we initially suspected that the model might struggle to learn the model-initiate paradigm. This paper therefore focuses primarily on whether the model can learn to proactively initiate inquiries when the user issues no query. We believe that a model capable of relying on streaming video rather than on user queries is highly meaningful. Regarding the metrics, we could not find official results on OVOBench and other datasets. We attempted to investigate this, but the metrics reported by OneStreamer also differ from those of MaLoW. This is mainly because OVOBench itself defines both strict and lenient protocols. In this paper, we consistently adopt a single configuration, which ensures the validity of our evaluation. As for the passive formulation, we will revise this description in the main text and adopt the established term proactive.
Q1: The choice of four types.
R: In our thought, binary and four types address different levels of questions. A Response/Silence binary decides whether to produce language at a given moment, but does not distinguish proactive intent confirmation, responses to received user input, and event notifications when no new question arises, behaviors with different triggers, interaction states, and supervision targets. For example, the same utterance ("Do you need any help?") may be a proactive inquiry or a response to the previous turn; a speak/silence label cannot express this functional difference. The four action types make these states explicit: Inquiry establishes helping intent, Response handles user input, Notify provides autonomous event assistance, and Silence preserves the option to keep observing. This refinement clarifies the types of model responses and better supports proactive questioning.
Q2: The human verification of human evaluation.
R: Our main focus is on designing a data pipeline akin to Flora. In our construction procedure, we first validate the effectiveness of the method at a small scale before proceeding to large-scale data construction. In future work, we will attempt to manually verify the entire dataset.
Q3: The clarification of windowing alleviates.
R: We wish to clarify this point. In Table 1, the Event-centric views are primarily intended to show the ratio of silence tokens to speak tokens in the windows obtained after cropping the video using the two Temporal windowing schemes. The full + event-centric views report the overall ratio after incorporating temporal windowing. Our aim is to show that data constructed with temporal windowing can substantially mitigate the imbalance in the silence-to-speak token ratio.
Q4: The human verification of the score and the choice of llm.
R: We adopt an LLM-as-a-judge approach primarily to accommodate the online training setting. Because it is difficult to predetermine when the model should ask and what it should ask, we need an LLM to serve as an immediate responder within the program rather than relying on manually specified replies. In Appendix A.5, we report manual evaluation results showing that LLM-as-a-judge achieved acceptable scores on most videos. We believe that a better-trained model could be used for this task in future work. As for the judge model, we selected Qwen3.8-27B and will state this in the main text (line 234). Semantic matching quality is determined by the LLM Judge, which assesses the semantic consistency between a model notification and the reference event, assigning scores of 0, 1, and 2 for mismatch, partial match, and sufficient match, respectively, which are then normalized to [0, 1].
Q5: The clarification of MII-Score on silence.
R: Actually, MII-Score does not ignore silence; rather, it implicitly evaluates silence decisions through event coverage and notification precision. Unnecessary Notify actions increase the precision denominator without yielding additional valid matching gains, whereas remaining silent when a reminder is warranted incurs a loss in event coverage. The metric therefore encourages speaking when appropriate and remaining silent when not, rather than separately rewarding the number of silent decisions, thereby avoiding the degenerate strategy of remaining silent throughout.