- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper addresses several limitations of existing streaming video assistants: their reliance on explicit requests, difficulty inferring useful assistance from high-level goals, and limited ability to track dynamic tasks. To address these challenges, The authors introduce StreamMind-Bench, covering explicit, implicit, and dynamic user intents. They train StreamMind-VL with SFT and localized trajectory RL, and use a StreamMind-Harness to maintain request state and interaction memory. Experiments show improvements in response timing and quality.
Critical strengths
S1: The motivation is clear and well justified.
Adapting to dynamic user requests is an important challenge for proactive video assistants, and this new benchmark dedicated to this problem is a meaningful contribution to the community.
S2: The proposed method combines policy learning with explicit management of user requests and interaction history.
The localized trajectory RL design is particularly useful for focusing optimization on the timing and quality of assistance at key moments. Strong results in full-trajectory evaluation, with substantial gains over several competing models, support the effectiveness of this approach.
Critical weaknesses
W1: Gap between the stated implicit-intent setting and the dataset examples.
Although the paper claims to be the first to study user intents in proactive streaming interaction and to handle broad, implicit requests such as “Help me keep an eye on this dish,” actual requests in the supplementary dataset can be much more explicit. For example, “Walk me through making this mug dessert, and catch me if I mix the ingredients out of order or mess up the measurements” specifies both the task and the desired assistance. Related goal-directed assistance has already been explored in VideoLLM-online [R1] and ProAssist [R2]. The authors should clarify what they mean by “implicit intent” and whether the dataset supports the claimed ability to infer specific assistance needs from broad requests.
W2: Unverified robustness of rule-based request updates.
The proposed StreamMind-Harness tracks dynamic requests through keywords and regular expressions, using cues such as “instead” to replace an old requirement. This may depend on specific wording: an equivalent request such as “I no longer want progress reports…” may fail to remove the old summary requirement. Although the original message is still passed to the LLM, the paper provides neither illustrative cases nor robustness experiments showing whether the system remains effective when alternative expressions are not captured by these rules.
References:
[R1] VideoLLM-online: Online Video Large Language Model for Streaming Video
[R2] Proactive Assistant Dialogue Generation from Streaming Egocentric Videos
Questions for authors
Q1: Scope of dynamic intent changes.
It seems that dynamic intent mainly add requirements or change assistance modes within the same task. Does the full benchmark include abandoning the original goal, such as changing from “help me cook” to “stop cooking and help me clean the kitchen”? Can the method generalize to such changes?
Q2: Recall loss after adding the Harness.
The paper attributes Harness gains to fewer unnecessary responses, but Dynamic RL recall drops from 52.58 to 30.65 in Table 13. Which necessary interventions are missed, and what causes this decline?
Q3: Request templates and rule development.
Is there a templates during the intent generation? What data were used to develop the Harness’s matching rules, and were evaluation requests consulted?
Q4: Inconsistent results across tables.
Table 2 reports RL Explicit Timing F1 of 26.51 and Implicit Content F1 of 19.13, versus 19.53 and 10.17 for RL+Harness in Table 13, while the Dynamic results agree. Please check whether the Explicit/Implicit results were swapped.
4 comments
rebuttal2
(Q1) Switching Task Goals in Dynamic Intents
Thank you for the comment. We would like to clarify that Revision (RV) includes not only adjustments to assistance methods within the same task, but also allows the original intent to be completely canceled and replaced with a new task goal.
(1) Switching task goals. StreamMind-Bench contains 500 Revision samples, with a total of 1,885 help evidence points. Existing examples mainly demonstrate adjustments to assistance goals within the same activity, but the definition of RV does not require the old and new intents to belong to the same task. For example, switching from “Help me cook” to “Stop helping me cook; help me clean the kitchen instead” also constitutes RV, because the original goal is canceled and replaced by the new one.
(2) Generalization capability of the method. StreamMind-Harness maintains an independent lifecycle for each intent. When the user explicitly switches tasks, the system can mark the old intent as invalid and register a new intent; the policy model then determines when to provide assistance based on the updated request and real-time visual environment. Therefore, by design, this mechanism does not depend on whether the old and new goals belong to the same activity, but instead adapts to task changes by continuously updating the valid intents.
(Q2) Analysis of Harness Recall and False-Trigger Rate
Thank you for the comment. We would like to emphasize that the main role of the Harness is to reduce redundant and unnecessary responses, rather than simply improve Recall. To further validate this, we analyze the effectiveness of model responses under Dynamic Intents, categorizing them into effective assistance (responses that meet current user needs and provide practical help) and ineffective responses (redundant, repetitive, or irrelevant content).
Table 9: Analysis of response content effectiveness for Dynamic Intents
| Response Type | RL | RL + Harness |
|---|---|---|
| Proportion of effective responses ↑ | 56.2% | 73.5% |
| Proportion of ineffective responses ↓ | 43.8% | 26.5% |
As shown by the experiment in Table 9, after introducing the Harness, the proportion of effective responses increases from 56.2% to 73.5%, while the proportion of ineffective responses decreases by 17.3 percentage points. The RL model exhibits higher Recall because it generates more unnecessary false-positive responses, but it also introduces a large number of redundant and ineffective responses. In contrast, the Harness effectively suppresses these unnecessary responses and improves overall interaction quality. Therefore, a decrease in Recall does not indicate a decrease in overall performance. For proactive interaction, we believe that Precision, FPR, and Content-F1 should be considered jointly to comprehensively evaluate response timing and content quality.
(Q4) Comparison of Results in Table 2 and Table 13
Thank you for the comment. We have updated and unified Table 13 and the related tables to ensure that all tables use consistent evaluation settings and result statistics, and to clearly distinguish the experimental configurations of StreamMind-VL and StreamMind-VL + Harness. The updated results further demonstrate the improvement in overall proactive interaction performance brought by the Harness. In particular, on Dynamic Intents, the Harness significantly reduces unnecessary responses and improves response timing and content quality. We have also updated the relevant tables and analyses in the revised version to make the performance comparison across different model configurations clearer.
Thank you for the detailed response. My main concerns have been addressed, and I will maintain my current rating.
Thank you for taking the time to review our rebuttal and for maintaining your positive score. We greatly appreciate your thoughtful feedback and your recognition of the additional evidence provided by our new analyses.
Rebuttal1
Thank you for your careful review and constructive comments, and for recognizing the research motivation, benchmark design, and methodological effectiveness of this work. We respond to your questions one by one below.
(1) Definition of implicit intents. In this work, implicit intents refer to cases where users express only a high-level task goal or a general need for assistance without explicitly specifying the particular assistance content and triggering conditions. For example, a user may make only an open-ended request, such as “Help me complete this task”, or further specify the type of assistance, such as “Walk me through making this mug dessert, and catch me if I mix the ingredients out of order”. Although the latter includes requests for guidance and error correction, it still does not specify exactly what guidance is needed, what errors may occur, or when intervention should take place. In both cases, the Agent must combine the user's overall needs, real-time environment state, and task progress to independently infer what assistance the user currently needs, when to provide it, and how to respond. Therefore, the key characteristic of implicit intents is not whether the request is sufficiently short or vague, but rather that the specific assistance needs must be determined through environment understanding and reasoning.
(2) Differences from VideoLLM-online and ProAssist. These works mainly focus on proactive assistance around predefined task goals, emphasizing timely responses for known objectives. In contrast, we focus on more open-ended proactive assistance scenarios: users may express only general needs, while the Agent must continuously observe the environment, independently identify potential opportunities for assistance, and decide when to intervene and what assistance to provide. Furthermore, this work covers various implicit needs in physical and digital environments, as well as dynamic revisions, compositions, and interruptions of user intents during interaction, bringing the task setting closer to that of general-purpose proactive intelligent assistants in real-world scenarios.
(W2 & Q3) Robustness of Harness Intent Updates
Thank you for the comment. We would like to clarify that the rules in the Harness only assist in maintaining intent states and do not directly control response decisions. The original user request is always provided to the policy model, allowing it to determine the user's latest needs based on semantics and the real-time environment.
To examine this, we design a semantically equivalent paraphrasing experiment on 1,500 Dynamic Intents samples, replacing keyword expressions such as “instead” with different phrasings, for example, “I no longer need progress reports; just flag mistakes”, while keeping the task semantics and visual content unchanged.
Table R7: Dynamic Intent Paraphrase Robustness (Trajectory with self-generated interaction history, ±2 s)
After semantic paraphrasing, RL + Harness shows a decrease in Timing F1 from 23.57 to 23.51 and in C-F1 from 11.02 to 10.98, while FPR decreases from 3.20 to 3.03. Similarly, for SFT + Harness, Timing F1 and C-F1 decrease by only 0.11 and 0.04 percentage points, respectively, and FPR increases by only 0.05 percentage points. Overall performance remains stable under both configurations, demonstrating the Harness's robustness to different expressions of user requests. Its response decisions do not rely solely on fixed keyword matching, but can combine the semantics of user instructions with interaction context to adapt to dynamically evolving user needs. We have included the corresponding experimental results and interaction examples in which rules are not matched in the revised version.