acceptodds
Under review as a conference paper at ICLR 2027

WatchWeave: Concurrent Proactive Perception and Transactional Action over Streaming Video

Abstract

Streaming video assistants must sustain user requests across ongoing observations, intervening questions, and instruction changes. Timely responses require tracking both visual events and the changing requests and reminder histories associated with them. We introduce WatchWeave, a runtime for concurrent proactive perception over a shared video stream. It maintains independently editable, versioned per-request records (Watches), combines event-phase and transition evidence with reminder history, and recovers weak event cues through relative changes in the model’s binary prediction scores. It checks proposed updates against current state and revalidates requests before releasing generated responses. On all 3,738 IPIBench episodes, WatchWeave achieves Overall scores of 30.58% and 34.54% with Qwen3-VL-8B and Qwen3-VL-32B, respectively. The 8B configuration exceeds published same-backbone IPI-Agent by 2.93 percentage points, with gains in repeated monitoring and multi-task management. On all 250 OmniInteract samples, native speech achieves an overall Interaction-Aware Quality-Timeliness F1 (IA-QTF1) score of 0.420 under video-time evaluation, extending the assessment to continuous multimodal interaction.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Streaming Video, Proactive Perception, Concurrent Task Management

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_5X6ROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes WatchWeave, a framework that maintains independent monitoring tasks, combines event-state information with notification history to determine when to respond, and validates task states before releasing responses. On IPIBench, WatchWeave achieves a 2.93 percentage-point improvement in overall performance over the published results of IPI-Agent using the same backbone model, with notable gains in repeated event monitoring and multi-task management.

Critical strengths
  1. Well-structured system design. WatchWeave integrates persistent task management, event-aware perception, and transactional execution into a unified framework. This provides a systematic approach to handling multiple concurrent user requests in streaming video scenarios.

  2. Promising empirical results. On IPIBench, WatchWeave improves overall performance by 2.93 percentage points over the published same-backbone IPI-Agent baseline, with larger gains in repeated event monitoring and multi-task management. The additional experiments on OmniInteract further demonstrate the framework's potential for multimodal interaction.

Critical weaknesses
  1. Insufficient reliability analysis of Relative-Peak Recovery. The method relies on relative Yes-token score fluctuations without adequately distinguishing meaningful event signals from numerical noise. Event-level false-positive analysis is lacking, and performance gains vary across tasks.

  2. Limited robustness of heuristic notification rules. The framework relies on manually specified confirmation thresholds and notification intervals, with limited sensitivity analysis. Mute-based rearming may also fail to distinguish new events from continuing activities.

  3. Insufficiently isolated ablations. Some ablations remove multiple dependent mechanisms simultaneously, making their individual contributions unclear. The benefits of phase–boundary consistency checks and transactional safeguards remain insufficiently validated.

  4. Limited real-time and generalization evaluation. Latency is evaluated on only nine episodes, without scalability analysis for increasing concurrent Watches. Generalization beyond Qwen-family backbones also remains untested.

  5. Presentation and terminology issues. Sections 3 and 3.1 ambiguously describe whether one request corresponds to one or multiple Watches. Citation formatting also needs clarification, particularly “Lu et al., 2026c, Sec. 3.3 and App. A.4” in Section 4.1 and “Li et al., 2026b, Tables 2, 4” in Table 2. Additionally, the main text leaves noticeable unused space on Page 9 rather than fully utilizing the permitted nine-page limit.

Questions for authors
  1. Relative-Peak reliability: How do the authors distinguish meaningful score rises from negligible fluctuations? Could event-level recovery precision and false-positive rates be provided?

  2. Notification policy: How does the controller distinguish new occurrences from continuing events after rearming? How sensitive are the results to confirmation thresholds and notification intervals?

  3. Component contributions: Could the authors provide targeted ablations isolating phase–boundary consistency, Registry persistence, and transactional safeguards such as CAS and Final Request Check?

  4. Runtime and generalization: How does latency scale with the number of concurrent Watches? Does the framework generalize to non-Qwen backbones?

6 comments
Reviewer_5X6ROpenReview ↗

Comments to the response

Thanks to the authors for the detailed rebuttal and additional experiments, which address my concerns regarding runtime scalability, generalization, and presentation.

However, several core concerns remain insufficiently addressed:

  1. Relative-Peak reliability: The six observed triggers (across 90,445 frames) are insufficient to establish reliability under numerical noise. The ±10% perturbation analysis only checks these six existing triggers, leaving false-positive rates and the sensitivity of non-triggering cases unaddressed.
  2. Notification policy robustness: Only 4–11 episodes per configuration (out of 3,738) entered the structured path under test. Thus, the small variation in aggregate scores reflects low coverage rather than robust policy behavior or reliable Mute-based event separation.
  3. Component contributions: Aside from the phase–boundary comparison combining a historical reference with mixed configurations, the 50-trial tests for Registry persistence, CAS, and Final Request Check yield only binary outcomes under controlled settings. They depart from Table 4's end-to-end ablation setup and lack quantitative benchmark-level evidence of individual contributions.

While I appreciate the authors' additional efforts, these responses do not sufficiently resolve my core concerns. I therefore maintain my original rating.

AuthorsOpenReview ↗

Response to follow-up concern 1

We apologize for the wait. Given the exceptionally large scale of the end-to-end benchmark parameter analysis, we used 128 H800 GPUs for 10 hours to conduct a full parameter analysis. Our detailed experimental results are presented below.

Thank you for further clarifying your remaining concerns and for acknowledging the additional evidence on runtime overhead, generalization, and presentation. We would like to further clarify the role of Relative-Peak and distinguish the conclusions supported by the notification policy experiments from those supported by the execution safeguard tests.

1. Relative-Peak is an optional compensatory recall mechanism, rather than a core dependency of WatchWeave.

We position Relative-Peak as an auxiliary mechanism for recovering notifications missed by the regular pathway. It is motivated by problems observed in practice: an event has already occurred, but the system remains silent or issues a notification too late. In the three cases presented, it recovered two missed notifications and reduced the delay of another notification from four seconds to one second. These specific scenarios demonstrate its practical value.

This discussion was originally included in the supplementary material. Since the mechanism proved effective in our actual system, we subsequently moved it to the main text to draw more attention to the approach. We present it in the main text to encourage researchers to consider this possibility for compensatory recall, rather than to make its general robustness to noise a prerequisite for WatchWeave's core contributions.

Its aggregate gains on the benchmark are relatively limited: in Table 4, the 8B and 32B models improve in Overall by 0.82 and 0.65 percentage points, respectively (given that the mechanism applies only to a subset of missed-notification scenarios, these gains still demonstrate its compensatory value). The regular pathway already covers many opportunities for proactive notification, and the compensatory mechanism primarily addresses the smaller number of remaining omissions. Its practical value in specific missed-notification scenarios therefore does not necessarily translate into a large aggregate score improvement.

Our intended conclusion is that the mechanism provides compensatory recall value in the real-system scenarios evaluated, rather than that it is always reliable under numerical noise. Sensitivity to perturbations is only a supplementary analysis for the current benchmark.

AuthorsOpenReview ↗

Response to follow-up concern 2

2. Notification policy: interpreting aggregate score changes through the relevant tasks.

These parameters are designed for specific notification behaviors: Act controls the confirmation threshold for the initial notification, Gap controls the notification interval, and Mute controls the restoration of notification eligibility after silence. In particular, suppressing and reissuing notifications for repeated events is directly related to Repeated Proactiveness; the nine tasks do not all involve these behaviors to the same extent. IPIBench is complex: Overall is the equally weighted average of the nine task scores, so local changes are diluted, and gains and losses across tasks can also offset one another.

Taking Repeated Proactiveness as an example: the default configuration scores 22.642%; Act = 4 scores 20.755%, a decrease of 1.887 percentage points; and Mute = 4 scores 21.226%, a decrease of 1.415 percentage points. Considered individually, these two changes contribute approximately −0.210 and −0.157 percentage points to Overall, respectively. After accounting for changes in the other tasks, the actual Overall decreases are 0.182 and 0.118 percentage points, respectively. Small aggregate fluctuations therefore do not mean that performance on the relevant tasks is unchanged.

Below are the original measurements for the seven configurations, each covering 3,738 episodes and 90,445 frames. The defaults are Act = 3, Gap = 6 seconds, and Mute = 3, with only one parameter changed at a time.

ConfigurationProactive TimingProactive UnderstandingRepeated ProactivenessMulti-task ManagementTask CancellationTask ModificationReactive after ProactiveReactive to ProactiveReactive under ProactiveOverall
Full system (default)50.70924.60222.64218.20347.22225.50330.82930.61928.45330.976
Act = 250.95224.60222.17017.96746.29625.50330.82930.61928.45330.821
Act = 450.95224.60220.75518.67646.75925.50330.82930.61928.45330.794
Gap = 4 seconds50.80024.60222.64218.91346.75925.16830.82930.61928.45330.976
Gap = 8 seconds50.86924.60221.69818.91346.75925.50330.82930.61928.45330.916
Mute = 250.95224.60221.69817.73048.14825.16830.82930.61928.45330.911
Mute = 450.96124.60221.22617.96747.22225.83930.82930.61928.45330.858

We further compared the original logs from the same set of runs episode by episode and counted the episodes that changed relative to the default configuration:

ConfigurationNotification count changedNotification timestamp sequence changedNotification timing or text changed
Act = 22357362
Act = 42560340
Gap = 4 seconds2650347
Gap = 8 seconds2756357
Mute = 22351342
Mute = 42457333
Reviewer_7UWtOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposed WatchWeave, a runtime for concurrent proactive perception over a shared video stream. It maintains independently editable, versioned perrequest records (Watches), combines event-phase and transition evidence with reminder history, and recovers weak event cues through relative changes in the model’s binary prediction scores. It checks proposed updates against current state and revalidates requests before releasing generated responses. Experiments show the effectiveness of the proposed model.

Critical strengths
  1. The paper extends proactive reminders from a single trigger to multiple concurrent requests, which is a strong and novel contribution.
Critical weaknesses
  1. The Controller contains a large number of hand-set parameters, and no sensitivity analysis was done on these parameters.
  2. The improvement from Relative-Peak is small and unstable.
Questions for authors
  1. Can you provide the experimental results of the parameter sensitivity analysis?
  2. Can you provide a comparison of the inference cost between WatchWeave and IPI-Agent?
Reviewer_uriTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces WatchWeave, a framework for addressing Concurrent Proactive Perception in streaming video. It aims to maintain multiple independently editable monitoring targets over a shared video stream while determining when to deliver appropriate and timely responses. The framework consists of three key components:

  • Persistent Intent Weaving: Decomposes user queries into independent Watches, each maintaining its own instruction versions, states, and notification history.
  • Occurrence-Centric Perception: Decouples event progression from notification eligibility, leveraging visual evidence and notification history to manage new and repeated notifications, with a recovery mechanism to compensate for missed event evidence.
  • Transactional Agency: Ensures state consistency and request validity to prevent conflicting updates and stale responses during concurrent execution.

Experiments on IPIBench and OmniInteract demonstrate the effectiveness of WatchWeave in repeated proactiveness, multi-task management, and other aspects of proactive interaction.

Critical strengths
  • Practical Problem and Framework Design: The paper addresses concurrent proactive monitoring in streaming video with a structured framework for managing evolving targets and delivering valid notifications.

  • Empirical Effectiveness: WatchWeave achieves improvements on IPIBench and OmniInteract, particularly in repeated proactiveness and multi-task management. The ablation studies further support the effectiveness of its key components.

Critical weaknesses
  • Limited Generality and Unclear Inference Efficiency: The framework primarily focuses on query-specified monitoring targets, while proactive streaming video interaction involves more diverse types of queries. Moreover, it is unclear whether each Watch requires separate model inference, and the additional computational overhead introduced by mechanisms such as Relative-Peak Recovery may affect real-time efficiency.

  • Insufficient Experimental Validation: The evaluation is primarily conducted on proactive interaction benchmarks, leaving generalization to broader streaming video understanding benchmarks such as OVO-Bench and StreamingBench unclear. Furthermore, comparisons with relevant state-of-the-art proactive methods are limited, making it difficult to fully assess the method's competitiveness.

Questions for authors
  • Does each Watch require a separate VLM inference with the video context?

  • Can the proposed framework be extended to streaming video tasks beyond predefined monitoring targets, such as those evaluated in OVO-Bench and StreamingBench?

  • Under what conditions does Relative-Peak Recovery provide meaningful improvements, and why, given its relatively small gains in the ablation studies?

Reviewer_htgwOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents WatchWeave, a framework for proactive video monitoring that maintains multiple user requests as a video stream unfolds. Each request is represented as an independently editable Watch with its own detection target, notification rules, and history, allowing unfinished tasks to persist across intervening questions and instruction changes. To determine when to respond, the framework combines event-phase and transition predictions with response proposals and prior notification history. It also uses relative peaks in binary prediction scores to recover weak event cues. A transactional execution mechanism checks proposed updates against the current task state and revalidates generated answers to suppress duplicate or outdated notifications. Experiments on IPIBench show improvements over the IPI-Agent baseline, particularly in repeated monitoring and multi-task management. The framework is further extended to Qwen3-Omni and evaluated on OmniInteract with text and native speech outputs under a video-time protocol.

Critical strengths

1. Explicit separation of event evidence and notification eligibility. WatchWeave makes a useful distinction between whether a target event is occurring and whether another notification is warranted given the current request and previous reminders. Combining event-phase and transition predictions with per-request histories provides a coherent mechanism for handling repeated events, while request validation addresses responses that become outdated during generation. This distinction is important for sustained proactive interaction, where correct behavior depends on both visual evidence and the evolving state of the user’s requests.

2. Empirical improvements aligned with the intended capabilities. With Qwen3-VL-8B, WatchWeave improves the IPIBench Overall score by 2.93 percentage points over the published same-backbone IPI-Agent baseline, with larger gains in Repeated Proactiveness (+5.66) and Multi-task Management (+5.20). The component ablations provide supporting evidence: removing event tracking substantially reduces repeated-monitoring performance, while disabling Registry updates particularly harms multi-task management. These results substantiate the value of maintaining explicit event and notification state for the sustained monitoring behaviors targeted by the paper.

Critical weaknesses

Limited evidence of generalization. WatchWeave is mainly evaluated on IPIBench and OmniInteract, its broader applicability remains unclear. Results on more benchmarks like OVO-Bench, StreamingBench, and SPOT-Bench would help establish whether the observed benefits generalize to a wider range of streaming video understanding tasks.

Questions for authors

See weakness, please.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal

We sincerely thank the Program Chairs, Area Chairs, and all reviewers for their time, effort, and constructive feedback. We are encouraged by the reviewers’ recognition of WatchWeave’s strengths and their engagement with our additional experiments. Our goal in this rebuttal revision is to strengthen the paper and improve confidence in its results.

Strengths recognized by the reviewers

  • Extending proactive monitoring to multiple concurrent requests (7UWt, uriT, 5X6R). WatchWeave supports independently editable monitoring targets and coordinates evolving user requests over a shared video stream.
  • Separating event evidence from notification eligibility (htgw). Explicit event state and reminder history distinguish whether an event is occurring from whether another notification is warranted.
  • Empirical gains aligned with sustained interaction (htgw, uriT, 5X6R). On IPIBench, WatchWeave improves Overall by 2.93 percentage points, Repeated Proactiveness by 5.66, and Multi-task Management by 5.20 over the published same-backbone IPI-Agent results, with additional evaluation on OmniInteract.

Questions raised and our responses

  • Broader task and backbone generalization (htgw, uriT, 5X6R). We clarified support for immediate questions and evolving monitoring goals, and added SPOT-Bench evaluations covering all ABD and PNR videos. WatchWeave improves T-F1@5 over our same-backbone baseline by 27.06/16.53 points on ABD/PNR. On all 3,738 IPIBench examples, it also improves InternVL3-9B Overall by 3.04 points without fine-tuning. We explained benchmark selection and documented the StreamingBench PO annotation issues identified in our audit.
  • Inference efficiency and concurrent-Watch scaling (uriT, 7UWt, 5X6R). We clarified that multiple Watches share a primary inference rather than requiring separate full-context inference per Watch. Increasing from two to eight Watches adds 14.5 ms to mean per-frame latency in the measured workload. We also explained why a matched IPI-Agent timing comparison was unavailable: its complete implementation was not publicly released at evaluation time.
  • Relative-Peak’s contribution and reliability (uriT, 7UWt, 5X6R). We provided on/off recovery examples, full-trace activation and reminder-alignment statistics, and score-perturbation analyses. These clarify its role as an auxiliary mechanism for recovering missed or late reminders and the scope of the reliability evidence.
  • Notification-policy sensitivity and event rearming (7UWt, 5X6R). We evaluated seven Controller configurations on the complete IPIBench, with Overall ranging from 30.79% to 30.98% across the tested settings. We also clarified that rearming restores notification eligibility; identifying a new occurrence additionally requires event evidence.
  • Individual component contributions and execution safeguards (5X6R). We reported phase/boundary policy results and added isolated tests of Registry persistence, stale-update rejection, and final request validation, explaining the distinct conclusions supported by benchmark results and targeted execution tests.

To this end, our final revision will include the following updates:

  • Clearer definitions of requests and Watches, shared inference, event evidence, notification eligibility, and Relative-Peak’s recovery role.
  • Expanded experimental evidence covering SPOT-Bench, InternVL3-9B, Controller sensitivity, concurrent-Watch latency, recovery analyses, and targeted safeguard tests.
  • Clearer presentation of evaluation protocols, published baseline sources, citation references, and the scope of each experimental analysis.

We hope these clarifications and additional results address the concerns raised and strengthen confidence in WatchWeave’s contribution to sustained, concurrent proactive video interaction. We sincerely appreciate the reviewers’ constructive engagement and welcome further questions and suggestions.

Thank you, the authors of submission #3339

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.