acceptodds
Under review as a conference paper at ICLR 2027

StreamMind: Proactive Video Agents for Evolving User Intent

10 views

Abstract

Vision-language models are shifting from offline passive understanding toward online proactive interaction. However, prevailing methods largely rely on explicit requests while miss the implicit user intent (e.g., “where should I click next?” while a user navigate a website), meanwhile fail to update the state simultaneously as user requests might update continuously. These limitations make them less suited to intent-aware interaction in practical. To address these challenges, we introduce **StreamMind-Bench**, a new benchmark serving as testbed for user intent interaction, with 4.1K evaluation samples and 80K training samples spanning explicit, implicit, and dynamic intents. Building on this benchmark, we develop **StreamMind-VL**, a streaming proactive model that infers implicit user intents from continuous observations and dynamically tailors its interactions as these intents evolve. We further introduce **StreamMind-Harness**, which maintains user intents as persistent states with a fleixble lifecycles via a hierarchical memory to track the latest user intent and environmental state. Our experiments reveal three key findings: *(i)* StreamMind-VL preserves strong general video understanding while improving response timing and answer quality by 10.7% over other baselines; *(ii)* existing proactive models remain limited in understanding user intent and deciding when to respond, especially under implicit and dynamic intents; and *(iii)* StreamMind-Harness further improves response consistency and adaptation over sustained interaction, resulting in a 12.3% gain in proactive interaction.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
proactive agent, streaming video, interaction, intent-aware

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_S5LkOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses several limitations of existing streaming video assistants: their reliance on explicit requests, difficulty inferring useful assistance from high-level goals, and limited ability to track dynamic tasks. To address these challenges, The authors introduce StreamMind-Bench, covering explicit, implicit, and dynamic user intents. They train StreamMind-VL with SFT and localized trajectory RL, and use a StreamMind-Harness to maintain request state and interaction memory. Experiments show improvements in response timing and quality.

Critical strengths

S1: The motivation is clear and well justified.

Adapting to dynamic user requests is an important challenge for proactive video assistants, and this new benchmark dedicated to this problem is a meaningful contribution to the community.

S2: The proposed method combines policy learning with explicit management of user requests and interaction history.

The localized trajectory RL design is particularly useful for focusing optimization on the timing and quality of assistance at key moments. Strong results in full-trajectory evaluation, with substantial gains over several competing models, support the effectiveness of this approach.

Critical weaknesses

W1: Gap between the stated implicit-intent setting and the dataset examples.

Although the paper claims to be the first to study user intents in proactive streaming interaction and to handle broad, implicit requests such as “Help me keep an eye on this dish,” actual requests in the supplementary dataset can be much more explicit. For example, “Walk me through making this mug dessert, and catch me if I mix the ingredients out of order or mess up the measurements” specifies both the task and the desired assistance. Related goal-directed assistance has already been explored in VideoLLM-online [R1] and ProAssist [R2]. The authors should clarify what they mean by “implicit intent” and whether the dataset supports the claimed ability to infer specific assistance needs from broad requests.

W2: Unverified robustness of rule-based request updates.

The proposed StreamMind-Harness tracks dynamic requests through keywords and regular expressions, using cues such as “instead” to replace an old requirement. This may depend on specific wording: an equivalent request such as “I no longer want progress reports…” may fail to remove the old summary requirement. Although the original message is still passed to the LLM, the paper provides neither illustrative cases nor robustness experiments showing whether the system remains effective when alternative expressions are not captured by these rules.

References:

[R1] VideoLLM-online: Online Video Large Language Model for Streaming Video

[R2] Proactive Assistant Dialogue Generation from Streaming Egocentric Videos

Questions for authors

Q1: Scope of dynamic intent changes.

It seems that dynamic intent mainly add requirements or change assistance modes within the same task. Does the full benchmark include abandoning the original goal, such as changing from “help me cook” to “stop cooking and help me clean the kitchen”? Can the method generalize to such changes?

Q2: Recall loss after adding the Harness.

The paper attributes Harness gains to fewer unnecessary responses, but Dynamic RL recall drops from 52.58 to 30.65 in Table 13. Which necessary interventions are missed, and what causes this decline?

Q3: Request templates and rule development.

Is there a templates during the intent generation? What data were used to develop the Harness’s matching rules, and were evaluation requests consulted?

Q4: Inconsistent results across tables.

Table 2 reports RL Explicit Timing F1 of 26.51 and Implicit Content F1 of 19.13, versus 19.53 and 10.17 for RL+Harness in Table 13, while the Dynamic results agree. Please check whether the Explicit/Implicit results were swapped.

4 comments

Thank you for your careful review and constructive comments, and for recognizing the research motivation, benchmark design, and methodological effectiveness of this work. We respond to your questions one by one below.

(1) Definition of implicit intents. In this work, implicit intents refer to cases where users express only a high-level task goal or a general need for assistance without explicitly specifying the particular assistance content and triggering conditions. For example, a user may make only an open-ended request, such as “Help me complete this task”, or further specify the type of assistance, such as “Walk me through making this mug dessert, and catch me if I mix the ingredients out of order”. Although the latter includes requests for guidance and error correction, it still does not specify exactly what guidance is needed, what errors may occur, or when intervention should take place. In both cases, the Agent must combine the user's overall needs, real-time environment state, and task progress to independently infer what assistance the user currently needs, when to provide it, and how to respond. Therefore, the key characteristic of implicit intents is not whether the request is sufficiently short or vague, but rather that the specific assistance needs must be determined through environment understanding and reasoning.

(2) Differences from VideoLLM-online and ProAssist. These works mainly focus on proactive assistance around predefined task goals, emphasizing timely responses for known objectives. In contrast, we focus on more open-ended proactive assistance scenarios: users may express only general needs, while the Agent must continuously observe the environment, independently identify potential opportunities for assistance, and decide when to intervene and what assistance to provide. Furthermore, this work covers various implicit needs in physical and digital environments, as well as dynamic revisions, compositions, and interruptions of user intents during interaction, bringing the task setting closer to that of general-purpose proactive intelligent assistants in real-world scenarios.

(W2 & Q3) Robustness of Harness Intent Updates

Thank you for the comment. We would like to clarify that the rules in the Harness only assist in maintaining intent states and do not directly control response decisions. The original user request is always provided to the policy model, allowing it to determine the user's latest needs based on semantics and the real-time environment.

To examine this, we design a semantically equivalent paraphrasing experiment on 1,500 Dynamic Intents samples, replacing keyword expressions such as “instead” with different phrasings, for example, “I no longer need progress reports; just flag mistakes”, while keeping the task semantics and visual content unchanged.

Table R7: Dynamic Intent Paraphrase Robustness (Trajectory with self-generated interaction history, ±2 s)

Checkpoint with HarnessRequestsTiming F1 ↑FPR ↓C-F1 ↑
RL + HarnessOriginal23.573.2011.02
RL + HarnessParaphrased23.513.0310.98
SFT + HarnessOriginal20.2110.338.50
SFT + HarnessParaphrased20.1010.388.46

After semantic paraphrasing, RL + Harness shows a decrease in Timing F1 from 23.57 to 23.51 and in C-F1 from 11.02 to 10.98, while FPR decreases from 3.20 to 3.03. Similarly, for SFT + Harness, Timing F1 and C-F1 decrease by only 0.11 and 0.04 percentage points, respectively, and FPR increases by only 0.05 percentage points. Overall performance remains stable under both configurations, demonstrating the Harness's robustness to different expressions of user requests. Its response decisions do not rely solely on fixed keyword matching, but can combine the semantics of user instructions with interaction context to adapt to dynamically evolving user needs. We have included the corresponding experimental results and interaction examples in which rules are not matched in the revised version.

(Q1) Switching Task Goals in Dynamic Intents

Thank you for the comment. We would like to clarify that Revision (RV) includes not only adjustments to assistance methods within the same task, but also allows the original intent to be completely canceled and replaced with a new task goal.

(1) Switching task goals. StreamMind-Bench contains 500 Revision samples, with a total of 1,885 help evidence points. Existing examples mainly demonstrate adjustments to assistance goals within the same activity, but the definition of RV does not require the old and new intents to belong to the same task. For example, switching from “Help me cook” to “Stop helping me cook; help me clean the kitchen instead” also constitutes RV, because the original goal is canceled and replaced by the new one.

(2) Generalization capability of the method. StreamMind-Harness maintains an independent lifecycle for each intent. When the user explicitly switches tasks, the system can mark the old intent as invalid and register a new intent; the policy model then determines when to provide assistance based on the updated request and real-time visual environment. Therefore, by design, this mechanism does not depend on whether the old and new goals belong to the same activity, but instead adapts to task changes by continuously updating the valid intents.

(Q2) Analysis of Harness Recall and False-Trigger Rate

Thank you for the comment. We would like to emphasize that the main role of the Harness is to reduce redundant and unnecessary responses, rather than simply improve Recall. To further validate this, we analyze the effectiveness of model responses under Dynamic Intents, categorizing them into effective assistance (responses that meet current user needs and provide practical help) and ineffective responses (redundant, repetitive, or irrelevant content).

Table 9: Analysis of response content effectiveness for Dynamic Intents

Response TypeRLRL + Harness
Proportion of effective responses ↑56.2%73.5%
Proportion of ineffective responses ↓43.8%26.5%

As shown by the experiment in Table 9, after introducing the Harness, the proportion of effective responses increases from 56.2% to 73.5%, while the proportion of ineffective responses decreases by 17.3 percentage points. The RL model exhibits higher Recall because it generates more unnecessary false-positive responses, but it also introduces a large number of redundant and ineffective responses. In contrast, the Harness effectively suppresses these unnecessary responses and improves overall interaction quality. Therefore, a decrease in Recall does not indicate a decrease in overall performance. For proactive interaction, we believe that Precision, FPR, and Content-F1 should be considered jointly to comprehensively evaluate response timing and content quality.

(Q4) Comparison of Results in Table 2 and Table 13

Thank you for the comment. We have updated and unified Table 13 and the related tables to ensure that all tables use consistent evaluation settings and result statistics, and to clearly distinguish the experimental configurations of StreamMind-VL and StreamMind-VL + Harness. The updated results further demonstrate the improvement in overall proactive interaction performance brought by the Harness. In particular, on Dynamic Intents, the Harness significantly reduces unnecessary responses and improves response timing and content quality. We have also updated the relevant tables and analyses in the revised version to make the performance comparison across different model configurations clearer.

Thank you for the detailed response. My main concerns have been addressed, and I will maintain my current rating.

Thank you for taking the time to review our rebuttal and for maintaining your positive score. We greatly appreciate your thoughtful feedback and your recognition of the additional evidence provided by our new analyses.

Reviewer_hjupOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces StreamMind-Bench, a benchmark for proactive interaction under explicit, implicit, and dynamic user intents. Building on this benchmark, the authors develop StreamMind-VL together with StreamMind-Harness, combining a streaming policy trained through SFT and RL with persistent intent tracking and hierarchical memory. Experiments evaluate response timing, answer quality, and generalization across the proposed and external benchmarks.

Critical strengths

StreamMind-Bench covers explicit, implicit, and dynamic user intents across physical and digital environments, providing a useful testbed for evaluating proactive assistance and adaptation to evolving user needs.

Critical weaknesses
  • The Explicit/Implicit results appear to be swapped between Tables 2 and 3. For example, Gemini-3.1-Pro’s F1 scores change from 65.28/77.56 to 77.56/65.28. Even after accounting for this swap, some values remain inconsistent: StreamMind-VL (RL)’s Explicit Recall/F1 in Table 2 are 64.89/60.88, whereas the corresponding Implicit values in Table 3 are 60.89/59.06. These discrepancies under the same stated evaluation setting raise serious concerns about the reliability of the reported results.

  • Trajectory evaluation is not clearly explained. Similarly, the description of the trajectory-level reward lacks sufficient detail.

  • Some symbols are not explicitly defined when introduced. For example, Section 3.1 includes H<t in the context ct without explaining its meaning there. The notation should be defined consistently at first use.

  • The main paper lacks implementation details. In particular, the base model used for StreamMind-VL is not specified, making the comparisons in Table 2 difficult to interpret.

Questions for authors
  • It is recommended to explain explicit/implicit swap and the remaining numerical inconsistencies between Tables 2 and 3, and clarify which results are correct.

  • Please clarify how trajectory evaluation is conducted and how the trajectory-level reward is computed.

10 comments

Thank you for your careful review. We also appreciate your recognition of the comprehensiveness of StreamMind-Bench in covering explicit, implicit, and dynamic user intents, as well as physical and digital environments, and its value in evaluating proactive assistance and adaptation to dynamic user needs. We respond to your comments point by point below.

(W1 & Q1) Verification of Table 2 and Appendix Results

Thank you for the comment. We have rechecked the experimental results in the main text and appendix and corrected the relevant tables and descriptions, as detailed below.

(1) Evaluation tolerances and table annotations. The captions of Tables 3–8 in the original appendix contained typographical errors, incorrectly labeling the evaluation tolerance as ±2s. In fact, these tables use ±4s and ±0s to analyze the effects of different temporal tolerances on evaluation results, whereas Table 2 in the main text uses ±2s. We have corrected the relevant table captions. Therefore, some numerical differences between the main text and appendix primarily arise from different evaluation tolerances, rather than inconsistencies in experimental results.

(2) Performance changes under a ±4s tolerance. When the temporal matching tolerance is relaxed, some correct responses with slight temporal offsets can be counted as valid matches, so the Content-F1 of most models increases. For example, Qwen3-VL-8B improves by 10.05 and 5.36 points on Explicit and Dynamic, respectively, while StreamMind-VL (SFT) improves by 3.37 and 5.28 points on Implicit and Dynamic, respectively. This indicates that evaluation tolerance affects temporal matching results and also shows that reporting performance under different tolerances helps provide a more comprehensive assessment of models' temporal sensitivity.

(3) Reasons for using Content-F1 as a comprehensive metric. Proactive assistance requires not only intervention at the right time, but also accurate and useful responses. Recall or Timing F1 alone cannot fully reflect this capability. For example, in Trajectory evaluation, JoyAI-VL-Interaction achieves a Recall as high as 95.44, but its Precision is only 4.67, indicating that frequent responses can cover more help points while introducing a large number of false triggers. In contrast, Content-F1 jointly considers temporal matching, response content quality, missed responses, and redundant responses, assigning scores only to successfully matched responses according to their semantic quality: $ \mathrm{C\text{-}F1}=\frac{2\sum s_{g,p}}{N_{\mathrm{pred}}+N_{\mathrm{gold}}} $

The experimental results show that StreamMind-VL performs better on Content-F1, indicating that our method can not only effectively identify appropriate intervention timing but also generate high-quality assistance content, demonstrating stronger overall capabilities in response timing and content quality.

(W2 & Q2) Trajectory Evaluation Procedure and Trajectory-Level Reward

Thank you for your suggestion. We further clarify the trajectory evaluation procedure and the computation of trajectory-level rewards and will add the corresponding explanations and pseudocode to Sections 3.5 and 4.2 of the revised version.

(1) Trajectory Evaluation

Trajectory evaluation consists of three main steps: (1) Continuous causal inference: the model processes the complete video at 1 FPS. At time tt, it accesses only historical video frames, user requests, and its own generated interaction history, without using future information or gold-standard interaction history, and independently decides between SILENCE/RESPONSE; (2) Temporal matching: predicted responses are matched one-to-one with gold-standard evidence windows using a ±2s temporal tolerance, based on which Precision, Recall, Timing F1, and FPR are calculated; (3) Content scoring: Claude Sonnet 4.6 is used to score the semantic quality of successfully matched responses, and ANS and Content-F1 are calculated while penalizing missed and redundant responses.

Unlike Pointwise Evaluation, which independently evaluates local decisions, Trajectory Evaluation relies on the model's self-generated interaction history throughout the entire process, enabling a more realistic assessment of challenges such as long-horizon error accumulation, dynamic intent tracking, and repeated or omitted responses.

(2) Trajectory-level Reward

During RL training, we construct localized causal trajectories of 10–15 seconds around annotated evidence points, sampling K=8K = 8 rollouts per prompt. Unlike the point-level reward RalignR_{\mathrm{align}}, which targets a single evidence point, the trajectory-level reward RglobR_{\mathrm{glob}} evaluates the response sequence of the entire rollout. Specifically, we use Claude Sonnet 4.6 with sample-specific scoring rubrics to assess five dimensions: (1) Coverage, whether necessary assistance is covered; (2) Density, whether the response frequency is appropriate; (3) Order, whether the response order follows the task logic; (4) Consistency, whether responses are consistent with the current user intent and historical responses; and (5) Redundancy, whether repeated or unnecessary assistance is present.

The scores for all five dimensions are normalized to [0, 1] and averaged to obtain the trajectory-level reward, which is then combined with the point-level reward through weighted aggregation: Rglob=15∑k=15rk,R=0.7Ralign+0.3RglobR_{\mathrm{glob}}=\frac{1}{5}\sum_{k=1}^{5}r_k,\qquad R=0.7R_{\mathrm{align}}+0.3R_{\mathrm{glob}}

GRPO then computes within-group relative advantages based on the combined rewards of the 8 rollouts under the same prompt and optimizes the model through a clipped policy objective and a KL regularization term.

(W3) Interaction History and Notation Definitions

Thank you for your suggestion. We further clarify the definition of interaction history H<tH_{<t}, which includes the agent's action decisions, generated response text, and user feedback before time tt. We will add this definition to Section 3.1 of the revised version and include a unified notation table in the appendix to improve clarity and consistency.

SymbolMeaning
V={vt}t=1TV=\{v_t\}_{t=1}^{T}Video stream sampled at 1 FPS
(ij,τj)(i_j,\tau_j)The jj-th user intent and its visible start time
(I≤t,At)(I_{\le t},\mathcal{A}_t)All intents up to time tt and the set of currently active intents
H<tH_{<t}Historical action decisions, response text, and user feedback
ct=⟨V≤t,I≤t,H<t⟩c_t=\langle V_{\le t},I_{\le t},H_{<t}\rangleCausal context at time tt
(at,yt)(a_t,y_t)Action decision and generated response text
(ξjk,[sjk,ejk],y^jk)(\xi_{jk},[s_{jk},e_{jk}],\hat y_{jk})Evidence point, valid response window, and reference response
St=⟨At,Pt,Rt⟩S_t=\langle\mathcal{A}_t,P_t,R_t\rangleStructured interaction state maintained by the Harness
(M<t,mt)(M_{<t},m_t)Hierarchical memory bank and retrieved relevant memory
c~t=ϕ(ct,St,mt)\tilde c_t=\phi(c_t,S_t,m_t)Enhanced context constructed by the Harness
(D,G)(\mathcal{D},\mathcal{G})Sets of decision tokens and response tokens in SFT

In addition, we will consistently use D\mathcal{D} and G\mathcal{G} to represent the two token sets in the SFT stage, avoiding notational ambiguity with the active intents At\mathcal{A}_t and RtR_t in the interaction state.

(W4) Base Model and Training Details

Thank you for the comment. StreamMind-VL is built upon Qwen3.5-9B (i.e., the reference model labeled "base model" in Figure 4 and Table 13). Therefore, the Qwen3.5-9B row in Table 2 is the direct comparison baseline for our method. We will explicitly state this base-model information in Section 5.1 of the revised main text and provide the complete training and implementation details in the appendix:

ItemConfiguration
Base modelQwen3.5-9B
Trainable parametersFreeze the vision encoder and jointly train the LLM and vision-language connector (Figure 3(b))
SFT data80K StreamMind task data + general streaming video data
SFT hyperparametersβ=0.9996\beta = 0.9996, λtext=2.5\lambda_{\text{text}} = 2.5, learning rate 1e−51\mathrm{e}{-5}, global batch size 608, 1 epoch (up to 1,385 steps), maximum input length 65,536 tokens
RL hyperparametersGRPO, 64 prompts per step, 8 rollouts per prompt, KL coefficient 0.05, learning rate 1e−61\mathrm{e}{-6}, default configuration of 500 steps
Video input1 FPS, dynamically scaled based on the pixel budget, with per-frame pixel limits of 65,536 / 32,768 for SFT/RL, respectively

These configurations have been added to the revised manuscript to improve experimental transparency and reproducibility.

Reviewer_YRNDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents StreamMind, a framework for proactive video agents that can infer implicit user intents and adapt to evolving requirements. It introduces StreamMind-Bench for evaluating different intent types, StreamMind-VL for optimizing response timing and content quality through SFT and RL, and StreamMind-Harness for maintaining interaction states over time.

Critical strengths
  1. Well-motivated research problem. The paper addresses an important and practical problem by extending proactive video interaction from explicit requests to implicit and evolving user intents.

  2. Comprehensive benchmark design. StreamMind-Bench covers diverse intent types and both physical and digital environments. Its pointwise and trajectory-level evaluation protocols provide a comprehensive testbed for proactive interaction.

  3. Extensive experimental analysis. The paper jointly considers response timing and content quality, with detailed experiments analyzing false triggers, interaction duration, and performance degradation over sustained interactions.

Critical weaknesses
  1. Insufficient ablations on the training strategies. More controlled comparisons would strengthen the paper, particularly: a. Adaptive weighting vs. standard SFT or fixed class weighting. b. Localized trajectory RL vs. full-trajectory RL under comparable training settings.

  2. Lack of real-world user evaluation. A user study would help verify whether the agent's proactive interventions align with actual user preferences, especially whether users genuinely want assistance at the annotated intervention moments.

Questions for authors

see the weakness

7 comments

Thank you for your constructive comments and for recognizing the motivation and practical significance of this work, the benchmark design covering multiple types of intents and both physical and digital environments, the two-level evaluation protocol, and the experimental analysis that considers both response timing and content quality. We respond to your comments point by point below.

(W1-a) Ablation of SFT Weighting Strategies

Thank you for your comment. We add a controlled ablation study in which all variants use the same base model, SFT data (70K task data + 768K general streaming video data), number of training steps, and hyperparameters, varying only the weighting strategy. Specifically, we compare four settings: (1) Standard SFT, where all token weights are set to 1; (2) Fixed Class Weighting, which uses fixed inverse-frequency class weights based on the global silence/response ratio of the entire mixed training set; (3) Adaptive Weighting (Ours), which dynamically updates the class weights γc(t)\gamma_c(t) according to Eq. (3)–(4), with β=0.9996\beta = 0.9996 and λtext=1.5\lambda_{\text{text}} = 1.5; and (4) Adaptive, λtext=1\lambda_{\text{text}} = 1, which further distinguishes the independent contributions of decision weighting and response-text weighting.

Table 1: Ablation of SFT weighting strategies (StreamMind-Bench, ±2s; averaged over the three intent types)

SFT VariantPW Timing F1 ↑PW FPR ↓PW C-F1 ↑Traj Timing F1 ↑Traj FPR ↓Traj C-F1 ↑
Standard SFT50.5765.5433.9910.0021.235.92
Fixed class weighting58.3460.7136.0312.3019.336.75
Adaptive, λtext=1\lambda_{\text{text}} = 160.0458.6038.7014.6718.547.03
Adaptive (Ours, λtext=1.5\lambda_{\text{text}} = 1.5)62.9856.1241.2020.3915.469.65

This design mainly considers that the silence/response distributions differ substantially across data sources, and that mixed sampling causes the class proportions to change dynamically during training. Standard SFT tends to produce excessive silence, while fixed class weighting has difficulty adapting to the dynamic distribution and may lead to over-response.

As shown in Table 1, compared with Standard SFT, our adaptive weighting improves Timing F1 by 12.41 points and reduces FPR by 9.42 points in Pointwise evaluation; in Trajectory evaluation, it improves Timing F1 by 10.39 points and reduces FPR by 5.77 points. Compared with Fixed Class Weighting, adaptive weighting further improves Timing F1 by 4.64/8.09 points and reduces FPR by 4.59/3.87 points under the two evaluation settings, respectively.

(W1-b) Comparison Between Localized Trajectory and Full-Trajectory RL

Thank you for your suggestion. We conduct an ablation study starting from the same SFT checkpoint under matched training budgets, keeping the number of GRPO update steps, the number of rollouts per prompt (K=8K = 8), the KL coefficient, and the total number of video frames processed per update consistent. Specifically, we compare four settings: (1) Full-trajectory RL, which performs causal rollouts over the full video stream and computes the same combined reward at the full-trajectory level; (2) Localized RL w/o RglobR_{\mathrm{glob}} (λ=1\lambda = 1), which uses only point-level alignment rewards; (3) Localized RL w/o RalignR_{\mathrm{align}} (λ=0\lambda = 0), which uses only trajectory-level rewards; and (4) Localized Trajectory RL (Ours, λ=0.7\lambda = 0.7), which constructs localized causal trajectories of approximately 10–15 seconds around annotated help points, with each trajectory containing only one scored evidence point, and optimizes with both point-level alignment and trajectory-level rewards.

Table 2: Ablation of RL strategies (StreamMind-Bench, Trajectory, ±2s)

VariantExp. F1 ↑Imp. F1 ↑Dyn. F1 ↑Avg FPR ↓Avg ANS ↑Avg C-F1 ↑
Full-trajectory RL19.3021.2411.455.4345.577.68
Localized RL w/o RglobR_{\mathrm{glob}} (λ=1\lambda = 1)20.7522.0913.083.3946.628.62
Localized RL w/o RalignR_{\mathrm{align}} (λ=0\lambda = 0)21.4023.5613.054.8746.308.96
Localized trajectory RL (Ours)26.5126.2619.963.0652.7712.96

The main reason for adopting localized trajectory RL is that most time steps in full-length videos involve silence decisions, while correct interventions, missed responses, and false positives are often mixed within the same trajectory, making a single trajectory-level reward prone to credit assignment diffusion. In contrast, localized trajectories concentrate the reward signal on response timing and content quality at critical moments, providing GRPO with clearer within-group relative advantage signals. As shown in Table 2, under the same training budget, compared with Full-trajectory RL, our method improves the average Timing F1 across the three intent types by 6.91 points, reduces the average FPR by 2.37 points, and improves Content-F1 by 5.28 points. In addition, compared with the variants using only point-level rewards or only trajectory-level rewards, the complete reward design improves Content-F1 by 4.34 and 4.00 points, respectively, further validating the complementary effects of localized trajectory optimization and the two reward types.

(W2) Real-User Study and Intervention Timing

Thank you for your suggestion. We have already conducted rigorous human validity verification during the original data construction process. As shown in Appendix Table 16, Temporal Validity requires interventions to occur when sufficient evidence is available and assistance remains useful, while Coverage and Non-redundancy ensure that unnecessary repeated interventions are avoided. All samples must receive unanimous approval from 4 independent reviewers. Across 6,005 candidate samples, inter-reviewer Fleiss' κ\kappa reaches 0.84, indicating high agreement among human reviewers on the annotations.

Furthermore, to directly validate the alignment between annotated intervention timing and real user needs, we design an independent user study. We stratify-sample 120 videos from 10 task subtypes and ask 5 participants who did not take part in data construction to watch the videos with the original annotations hidden and mark the moments when they would expect the assistant to intervene. At the same time, through anonymized interaction replays, they provide subjective evaluations of the helpfulness, timing appropriateness, and degree of interruption of different models.

Table 3: User study results

(a) Agreement between annotated intervention timing and real user needs

MetricExplicitImplicitDynamic
Proportion of user-marked points falling within annotated windows ↑88.3%79.2%75.6%
Proportion of annotated windows approved by a majority of users ↑85.4%76.8%72.5%
Inter-user agreement, Fleiss' κ\kappa ↑0.790.710.67

(b) Subjective experience scores (1–5)

ModelHelpfulness ↑Timing appropriateness ↑Degree of interruption ↓
Gemini-3.8-Flash3.423.312.65
MMDuet23.583.472.42
StreamMind-VL (RL)4.124.081.83
StreamMind-VL (RL) + Harness4.384.311.56

These results show that the benchmark's annotated intervention timing is highly consistent with real user needs. Meanwhile, StreamMind-VL performs better in terms of helpfulness, timing appropriateness, and reducing unnecessary interruptions, further validating the benchmark's applicability to real users and the effectiveness of the model's proactive interaction.

Reviewer_YRNDOpenReview ↗

comment

The authors' response addresses most of my concerns. However, the results suggest that the proposed model is highly sensitive to the hyperparameter (\lambda_{\text{text}}). Specifically, reducing (\lambda_{\text{text}}) from 1.5 to 1.0 leads to a consistent degradation in performance across all evaluation metrics. For example, Traj Timing F1 drops substantially from 20.39 to 14.67, while PW Timing F1 decreases from 62.98 to 60.04. Meanwhile, both PW EPR and Traj EPR increase, indicating further performance degradation. This raises concerns about the robustness of the proposed method to hyperparameter selection. The authors are encouraged to provide further justification for choosing (\lambda_{\text{text}}=1.5) and clarify whether the model can maintain stable performance under different parameter settings.

Comments on the paper

1 comment

We sincerely thank the three reviewers for their careful reviews and constructive comments, as well as for recognizing the main contributions of this work. The reviewers acknowledged the clear and practically valuable research motivation of extending proactive video interaction from explicit requests to implicit needs and dynamically evolving user intents (YRND, S5Lk); recognized the systematic and comprehensive design of StreamMind-Bench, which jointly covers three types of user intents and both physical and digital environments, and evaluates response timing and content quality through two levels of evaluation, Pointwise and Trajectory (YRND, hjup, S5Lk); and also appreciated the comprehensive experimental analysis and effective methodological design, particularly the value of localized trajectory reinforcement learning and persistent intent-state management for proactive interaction (YRND, S5Lk).

To address the reviewers' main concerns, we have further improved the experimental analysis and methodological descriptions in the following four aspects:

  1. Training strategy ablations (YRND W1). We add controlled comparisons of the SFT adaptive weighting strategy, localized trajectory RL, and full-trajectory RL, as well as ablations of different reward components, to further validate the effectiveness of each design.
  2. Real-user study (YRND W2). We invite 5 independent participants to annotate intervention timing and provide subjective evaluations on 120 videos, validating the consistency between benchmark annotations and real user needs, and assessing the helpfulness and degree of interruption of the model.
  3. Verification of experimental results and implementation details (hjup W1–W4, S5Lk Q4). We unify the evaluation results and table descriptions in the main text and appendix, further clarify Content-F1, the trajectory evaluation procedure, and reward design, and provide additional details on model architecture, notation definitions, and training configurations.
  4. Intent definitions and Harness analysis (S5Lk W1–W2, Q1–Q3). We clarify the definitions of implicit and dynamic intents and provide additional analyses of the Harness's robustness to semantic paraphrasing and response effectiveness. Experiments show that the Harness effectively reduces redundant responses and increases the proportion of useful assistance, thereby improving overall proactive interaction quality.

Below, we respond to the reviewers' comments point by point and include the corresponding experimental results and explanations in the revised version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.