acceptodds
Withdrawn from ICLR 2027

Stay Silent, Warn Timely: Persistent Video Agents for Online Anomaly Detection

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

In safety-critical applications, anomalies must be detected from incoming video in time to support intervention. Recent video-language interaction models can process streaming observations and decide when to respond, offering a founda- tion for persistent video anomaly detection (VAD). Yet online VAD is still com- monly evaluated through short-clip scores or isolated alarms, leaving the session- level behavior required for continuous monitoring largely unmeasured. A practi- cal monitor must maintain one query, remain silent during normal activity, issue timely evidence-grounded warnings, and continue after responding. We formu- late this setting as Timely Anomaly Interaction, in which a model observes only the available video prefix and produces a trajectory of silence and response de- cisions. We introduce TAI-VAD, a benchmark and training set covering single- event, multi-event, and long-video sessions. It provides temporal anomaly inter- vals and timestamp-aligned causal descriptions, and evaluates warning timeliness, anomaly coverage, normal-time false alarms, and post-response monitoring. Ex- periments with open video interaction models reveal that general interaction abil- ity does not reliably transfer to persistent anomaly monitoring: greater anomaly sensitivity often coincides with more false alarms, while conservative models miss events. To establish a task-specific baseline, we propose PACT, which a super- vised fine-tuning method to adapt a pretrained interaction model. PACT achieves the highest Single-event AUROC (0.711) and improves later-event recall over its backbone in Multi-event sessions. Together, our task formulation, benchmark, and training baseline enable systematic study of persistent video anomaly monitoring.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
video anomaly detection, vision-language model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_NAKkOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces TAI-VAD, a benchmark for persistent video anomaly monitoring. Unlike conventional clip-level evaluation, it evaluates how models behave over complete video sessions, including when to stay silent, when to issue warnings, and whether they can detect subsequent anomalies. The benchmark covers single-event, multi-event, and long-video settings. The authors also propose PACT, which fine-tunes an existing video interaction model using weighted trajectory-level supervision. Experiments show that current video interaction models struggle to balance anomaly detection and false alarms. PACT improves performance in the single-event setting, but its results on multi-event and long-video monitoring remain limited.

Critical strengths
  • The paper addresses a practical problem. Evaluating anomaly detection over complete video sessions, rather than isolated clips, makes sense for real-world monitoring applications where both timely detection and false alarms matter.

  • The benchmark brings together several existing datasets and provides a unified evaluation protocol for single-event, multi-event, and long-video scenarios. This could be a useful resource for evaluating streaming video models.

  • The experiments cover a reasonable range of video interaction models and reveal some interesting limitations, particularly the trade-off between anomaly sensitivity and false alarms. The ablation on loss weighting is also useful for understanding PACT's behavior.

Critical weaknesses
  • I am not convinced that the proposed task is fundamentally different from conventional online video anomaly detection. Existing spatio-temporal models can continuously process incoming video windows and produce binary anomaly predictions, much like online time-series anomaly detection. They can naturally handle multiple events and prolonged normal periods without requiring a new interaction formulation. Since the evaluation ultimately relies on binary silence/response decisions, it is unclear what additional capability this task actually requires.

  • Related to the above, the paper does not compare against conventional online VAD methods. Most baselines are video-language interaction models, even though standard causal or streaming VAD models could be evaluated under the same protocol. Without these comparisons, it is hard to tell whether the proposed approach offers any real advantage over existing detection methods.

  • The novelty of the semantic description component is also unclear. Previous works, such as LAVAD and Holmes-VAD, have already explored using visual descriptions and language models for video anomaly detection and understanding. While this paper emphasizes timestamp-aligned causal descriptions, their specific contribution is not isolated through ablations. More importantly, the evaluation only considers when warnings are generated, not whether their descriptions are accurate or grounded in the video.

  • PACT itself seems fairly straightforward, mainly applying weighted SFT to an existing interaction model. There is little methodological novelty beyond the training setup. Its effectiveness is also questionable in the more challenging settings: the multi-event false-positive rate is 55.08%, and long-video Recall@5s is only 3.28%. These results do not provide strong evidence that PACT successfully addresses the persistent monitoring problem it sets out to solve.

  • I also have some concerns about the evaluation design. The reported Binary AUROC is computed from binary actions and is effectively equivalent to balanced accuracy, rather than the usual AUROC based on continuous anomaly scores. Also, the coverage metric rewards repeated responses throughout an anomaly interval, although repeatedly warning about the same event may not be desirable in practice.

Overall, my main concern is that the paper does not clearly establish or separately validate the novelty of its three main components: the problem formulation, the semantic descriptions, and the PACT learning strategy. Each seems closely related to existing approaches, and the experiments do not sufficiently clarify what each component contributes beyond prior work.

Questions for authors
  1. What fundamentally distinguishes the proposed persistent monitoring task from conventional online video anomaly detection? Since existing causal spatio-temporal models can continuously produce binary predictions over video streams, it is unclear what additional capability the proposed formulation requires.

  2. Could the authors include conventional online or streaming VAD baselines under the same evaluation protocol? This seems important for understanding whether language-based interaction actually provides an advantage over existing detection approaches.

  3. What is the specific contribution of the timestamp-aligned causal descriptions compared to existing caption-based VAD approaches? An ablation without these descriptions would help clarify their effectiveness. Also, why is the factual grounding of generated warnings not evaluated?

  4. Given the high false-positive rate in the multi-event setting and the very low recall in the long-video setting, to what extent does PACT actually improve persistent monitoring? Also, does the current coverage metric appropriately reflect useful monitoring behavior, considering that repeatedly warning about the same event can increase the score?

Reviewer_ruYqOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies online video anomaly detection as a persistent monitoring task: a model observes only the video seen so far, stays silent during normal activity, warns about anomalies, and continues monitoring after a response. Its main contributions are TAI-VAD, a benchmark assembled from 4,953 existing videos with single-event, multi-event, and long-video test tracks, and PACT, a weighted supervised fine-tuning baseline built on JoyAI-VL-Interaction.

Validation compares video and streaming interaction models on the three tracks using warning timing, anomaly coverage, and normal-time false responses. PACT achieves the best single-event binary-action AUROC (0.711) and improves later-segment recall over its backbone. Its multi-event false-response rate remains high, however, and long-video timely recall is only 3.28%. The reported evaluation measures when models respond; it does not assess whether their warning text is factually supported by the video.

Critical strengths
  1. A consequential benchmark for persistent monitoring. TAI-VAD evaluates complete, causally observed video sessions across single-event, multi-event, and long-video settings, including normal intervals and responses to later segments. This matters because it exposes false alarms and missed later events that short-clip or first-alarm evaluations can miss.

  2. A substantive empirical baseline and diagnostic evaluation. On the single-event track, PACT improves binary-action AUROC from 0.640 for the strongest pretrained baseline to 0.711, while improving timely recall and reducing normal-time false responses. The paper also reports its poor long-video results openly. Together, these results show that the benchmark can distinguish useful progress from persistent failure modes, even though PACT does not solve the full task.

Critical weaknesses
  1. PACT's improvement on Single-event videos does not carry over to the Long-video track, which is central to the paper's motivation. Table 12 reports Recall@5s of 3.28% (4 of 122 segments), Coverage of 0.29%, and normal-time FPR of 8.88%. Under Eq. (6), this gives a binary AUROC of about 0.457, below the 0.5 of an always-silent policy on that metric. VideoChat3 achieves higher recall and coverage with lower FPR on the same track. The Multi-event result is also a difficult trade-off: 55.07% later-segment recall comes with 55.08% normal-time FPR. The paper reports these limitations, but the evidence for PACT as a persistent monitoring method remains weak.

  2. The evaluation does not establish that the warnings are grounded in visual evidence. The main text infers grounding from the fact that the model sees only the available video prefix (p. 5, lines 244–245). The appendix, however, states that the reported metrics score the timing of a RESPONSE, not the factual content of its text. A warning that identifies the wrong incident would still receive detection credit. The claim should be limited to response timing unless the authors assess warning content against the video prefix available at the time.

  3. The baseline set leaves out methods that directly address online VAD. REWARD, MoniTor, and AssistPDA appear in the related work but not in Tables 1–3; the paper also does not discuss ECCV 2026 ReactVAU. Because the main metrics concern actions and timing, score-based online detectors could be evaluated with an explicit causal time-alignment and threshold-calibration protocol. Their published AUCs are not directly comparable to the numbers here, but omitting them leaves an important question open: does PACT improve on online anomaly detectors, or mainly on the selected, largely unadapted interaction models?

  4. The two-second matching tolerance can credit an alarm issued before the annotated onset. Such a response may be assigned to the anomaly, while nearby normal checkpoints are excluded from FPR (p. 19, lines 987–1004). PACT's median matched delay on Multi-event videos is −2 seconds, exactly the earliest permitted boundary, and its normal-time FPR is 55.08%. The early responses might reflect visible precursors or uncertain annotations; aggregate results cannot distinguish those cases from premature alarms. Results with 0-, 1-, and 2-second tolerances, preferably at a common false-alarm budget, would make the timeliness claim easier to assess.

  5. The discussion points to memory dilution as an explanation for the Long-video failure, but the experiments do not separate it from other factors. The paper acknowledges a possible source-domain difference between abnormal and normal Long-video groups (p. 18). VideoChat3 also uses a different query policy on Single versus Multi/Long (p. 21), while PACT is trained on short windows. None of these factors is held fixed in a memory ablation. The text on p. 20 refers to results stratified by video duration and time to anomaly onset, but I could not find those results in the supplied PDF. Memory dilution is a plausible explanation; the current evidence does not identify it as the cause.

Questions for authors
  1. After interval merging, how many of the 113 scored Multi-event videos contain at least two segments? How many of those have been verified to contain distinct events, rather than stages of one event? Please report Later@5s and All@5s on that subset, alongside later recall conditional on a timely first warning.

  2. Page 20 states that Long-video results are stratified by video duration and time to anomaly onset. Where are those results reported, and do they support the proposed memory-based explanation after accounting for differences in video source?

  3. What proportion of matched warning texts makes a factual claim unsupported by the video prefix available at the response time? Without a prefix-only content assessment, the reported action metrics cannot establish that warnings are evidence-grounded.

  4. How sensitive are Recall@5s and response delay to the two-second boundary tolerance? Results with zero- and one-second tolerances are particularly relevant because PACT’s median matched delay on Multi-event videos is −2 seconds.

  5. How does PACT compare with online-VAD methods such as MoniTor and ReactVAU under the same causal replay protocol and a matched false-alarm budget? If a faithful adaptation is infeasible, what specific part of their output or timing protocol prevents it?

Reviewer_XntNOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Timely Anomaly Interaction and the TAI-VAD benchmark for persistent online video anomaly monitoring. It evaluates timely warnings, false alarms, and continued monitoring across single-event, multi-event, and long-video scenarios. The proposed PACT method adapts a video interaction model through trajectory-level supervised fine-tuning.

Critical strengths

The persistent monitoring formulation addresses an important gap in conventional video anomaly detection evaluation.

TAI-VAD provides a useful benchmark covering response timeliness, false alarms, and post-warning monitoring.

Critical weaknesses

PACT performs poorly in long-video monitoring, achieving only 3.28% timely recall and 0.29% anomaly coverage, substantially limiting its practical effectiveness.

The proposed training method is largely standard supervised fine-tuning, with limited methodological novelty beyond the task formulation.

Questions for authors

What causes the severe performance degradation of PACT on long-video sessions?

Can the authors compare PACT against simpler supervised fine-tuning baselines under identical training and evaluation settings?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.