- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes WatchWeave, a framework that maintains independent monitoring tasks, combines event-state information with notification history to determine when to respond, and validates task states before releasing responses. On IPIBench, WatchWeave achieves a 2.93 percentage-point improvement in overall performance over the published results of IPI-Agent using the same backbone model, with notable gains in repeated event monitoring and multi-task management.
Critical strengths
-
Well-structured system design. WatchWeave integrates persistent task management, event-aware perception, and transactional execution into a unified framework. This provides a systematic approach to handling multiple concurrent user requests in streaming video scenarios.
-
Promising empirical results. On IPIBench, WatchWeave improves overall performance by 2.93 percentage points over the published same-backbone IPI-Agent baseline, with larger gains in repeated event monitoring and multi-task management. The additional experiments on OmniInteract further demonstrate the framework's potential for multimodal interaction.
Critical weaknesses
-
Insufficient reliability analysis of Relative-Peak Recovery. The method relies on relative Yes-token score fluctuations without adequately distinguishing meaningful event signals from numerical noise. Event-level false-positive analysis is lacking, and performance gains vary across tasks.
-
Limited robustness of heuristic notification rules. The framework relies on manually specified confirmation thresholds and notification intervals, with limited sensitivity analysis. Mute-based rearming may also fail to distinguish new events from continuing activities.
-
Insufficiently isolated ablations. Some ablations remove multiple dependent mechanisms simultaneously, making their individual contributions unclear. The benefits of phase–boundary consistency checks and transactional safeguards remain insufficiently validated.
-
Limited real-time and generalization evaluation. Latency is evaluated on only nine episodes, without scalability analysis for increasing concurrent Watches. Generalization beyond Qwen-family backbones also remains untested.
-
Presentation and terminology issues. Sections 3 and 3.1 ambiguously describe whether one request corresponds to one or multiple Watches. Citation formatting also needs clarification, particularly “Lu et al., 2026c, Sec. 3.3 and App. A.4” in Section 4.1 and “Li et al., 2026b, Tables 2, 4” in Table 2. Additionally, the main text leaves noticeable unused space on Page 9 rather than fully utilizing the permitted nine-page limit.
Questions for authors
-
Relative-Peak reliability: How do the authors distinguish meaningful score rises from negligible fluctuations? Could event-level recovery precision and false-positive rates be provided?
-
Notification policy: How does the controller distinguish new occurrences from continuing events after rearming? How sensitive are the results to confirmation thresholds and notification intervals?
-
Component contributions: Could the authors provide targeted ablations isolating phase–boundary consistency, Registry persistence, and transactional safeguards such as CAS and Final Request Check?
-
Runtime and generalization: How does latency scale with the number of concurrent Watches? Does the framework generalize to non-Qwen backbones?
6 comments
Response to follow-up concern 1
We apologize for the wait. Given the exceptionally large scale of the end-to-end benchmark parameter analysis, we used 128 H800 GPUs for 10 hours to conduct a full parameter analysis. Our detailed experimental results are presented below.
Thank you for further clarifying your remaining concerns and for acknowledging the additional evidence on runtime overhead, generalization, and presentation. We would like to further clarify the role of Relative-Peak and distinguish the conclusions supported by the notification policy experiments from those supported by the execution safeguard tests.
1. Relative-Peak is an optional compensatory recall mechanism, rather than a core dependency of WatchWeave.
We position Relative-Peak as an auxiliary mechanism for recovering notifications missed by the regular pathway. It is motivated by problems observed in practice: an event has already occurred, but the system remains silent or issues a notification too late. In the three cases presented, it recovered two missed notifications and reduced the delay of another notification from four seconds to one second. These specific scenarios demonstrate its practical value.
This discussion was originally included in the supplementary material. Since the mechanism proved effective in our actual system, we subsequently moved it to the main text to draw more attention to the approach. We present it in the main text to encourage researchers to consider this possibility for compensatory recall, rather than to make its general robustness to noise a prerequisite for WatchWeave's core contributions.
Its aggregate gains on the benchmark are relatively limited: in Table 4, the 8B and 32B models improve in Overall by 0.82 and 0.65 percentage points, respectively (given that the mechanism applies only to a subset of missed-notification scenarios, these gains still demonstrate its compensatory value). The regular pathway already covers many opportunities for proactive notification, and the compensatory mechanism primarily addresses the smaller number of remaining omissions. Its practical value in specific missed-notification scenarios therefore does not necessarily translate into a large aggregate score improvement.
Our intended conclusion is that the mechanism provides compensatory recall value in the real-system scenarios evaluated, rather than that it is always reliable under numerical noise. Sensitivity to perturbations is only a supplementary analysis for the current benchmark.
Response to follow-up concern 2
2. Notification policy: interpreting aggregate score changes through the relevant tasks.
These parameters are designed for specific notification behaviors: Act controls the confirmation threshold for the initial notification, Gap controls the notification interval, and Mute controls the restoration of notification eligibility after silence. In particular, suppressing and reissuing notifications for repeated events is directly related to Repeated Proactiveness; the nine tasks do not all involve these behaviors to the same extent. IPIBench is complex: Overall is the equally weighted average of the nine task scores, so local changes are diluted, and gains and losses across tasks can also offset one another.
Taking Repeated Proactiveness as an example: the default configuration scores 22.642%; Act = 4 scores 20.755%, a decrease of 1.887 percentage points; and Mute = 4 scores 21.226%, a decrease of 1.415 percentage points. Considered individually, these two changes contribute approximately −0.210 and −0.157 percentage points to Overall, respectively. After accounting for changes in the other tasks, the actual Overall decreases are 0.182 and 0.118 percentage points, respectively. Small aggregate fluctuations therefore do not mean that performance on the relevant tasks is unchanged.
Below are the original measurements for the seven configurations, each covering 3,738 episodes and 90,445 frames. The defaults are Act = 3, Gap = 6 seconds, and Mute = 3, with only one parameter changed at a time.
| Configuration | Proactive Timing | Proactive Understanding | Repeated Proactiveness | Multi-task Management | Task Cancellation | Task Modification | Reactive after Proactive | Reactive to Proactive | Reactive under Proactive | Overall |
|---|---|---|---|---|---|---|---|---|---|---|
| Full system (default) | 50.709 | 24.602 | 22.642 | 18.203 | 47.222 | 25.503 | 30.829 | 30.619 | 28.453 | 30.976 |
| Act = 2 | 50.952 | 24.602 | 22.170 | 17.967 | 46.296 | 25.503 | 30.829 | 30.619 | 28.453 | 30.821 |
| Act = 4 | 50.952 | 24.602 | 20.755 | 18.676 | 46.759 | 25.503 | 30.829 | 30.619 | 28.453 | 30.794 |
| Gap = 4 seconds | 50.800 | 24.602 | 22.642 | 18.913 | 46.759 | 25.168 | 30.829 | 30.619 | 28.453 | 30.976 |
| Gap = 8 seconds | 50.869 | 24.602 | 21.698 | 18.913 | 46.759 | 25.503 | 30.829 | 30.619 | 28.453 | 30.916 |
| Mute = 2 | 50.952 | 24.602 | 21.698 | 17.730 | 48.148 | 25.168 | 30.829 | 30.619 | 28.453 | 30.911 |
| Mute = 4 | 50.961 | 24.602 | 21.226 | 17.967 | 47.222 | 25.839 | 30.829 | 30.619 | 28.453 | 30.858 |
We further compared the original logs from the same set of runs episode by episode and counted the episodes that changed relative to the default configuration:
| Configuration | Notification count changed | Notification timestamp sequence changed | Notification timing or text changed |
|---|---|---|---|
| Act = 2 | 23 | 57 | 362 |
| Act = 4 | 25 | 60 | 340 |
| Gap = 4 seconds | 26 | 50 | 347 |
| Gap = 8 seconds | 27 | 56 | 357 |
| Mute = 2 | 23 | 51 | 342 |
| Mute = 4 | 24 | 57 | 333 |
Comments to the response
Thanks to the authors for the detailed rebuttal and additional experiments, which address my concerns regarding runtime scalability, generalization, and presentation.
However, several core concerns remain insufficiently addressed:
While I appreciate the authors' additional efforts, these responses do not sufficiently resolve my core concerns. I therefore maintain my original rating.