Stay Silent, Warn Timely: Persistent Video Agents for Online Anomaly Detection

In safety-critical applications, anomalies must be detected from incoming video in time to support intervention. Recent video-language interaction models can process streaming observations and decide when to respond, offering a founda- tion for persistent video anomaly detection (VAD). Yet online VAD is still com- monly evaluated through short-clip scores or isolated alarms, leaving the session- level behavior required for continuous monitoring largely unmeasured. A practi- cal monitor must maintain one query, remain silent during normal activity, issue timely evidence-grounded warnings, and continue after responding. We formu- late this setting as Timely Anomaly Interaction, in which a model observes only the available video prefix and produces a trajectory of silence and response de- cisions. We introduce TAI-VAD, a benchmark and training set covering single- event, multi-event, and long-video sessions. It provides temporal anomaly inter- vals and timestamp-aligned causal descriptions, and evaluates warning timeliness, anomaly coverage, normal-time false alarms, and post-response monitoring. Ex- periments with open video interaction models reveal that general interaction abil- ity does not reliably transfer to persistent anomaly monitoring: greater anomaly sensitivity often coincides with more false alarms, while conservative models miss events. To establish a task-specific baseline, we propose PACT, which a super- vised fine-tuning method to adapt a pretrained interaction model. PACT achieves the highest Single-event AUROC (0.711) and improves later-event recall over its backbone in Multi-event sessions. Together, our task formulation, benchmark, and training baseline enable systematic study of persistent video anomaly monitoring.