- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper focuses on VLMs’ detection of physically implausible events in generated videos, including their number and temporal localization. It introduces VIOLATE, containing 556 synthetic videos and 1,552 human-reviewed event annotations, and uses 150 real videos to evaluate false positives. The benchmark supports two tasks: determining whether a given event description and time interval are valid, and discovering all anomalous events based solely on the video. The detector adopts three training stages: using human event lists to assist a teacher in generating supervision signals, reinforcement learning based on human reference events, and self-distillation combining the model’s own predictions with empty-event samples. The paper validates its contributions through comparisons across multiple models, experiments across training stages, an initialization ablation, a video-level filtering cascade, a verifier-reward comparison, and qualitative examples. The existing results support the challenging nature of the task and demonstrate the potential of the training pipeline, but evaluation reliability and the interpretation of no-event recognition capabilities require further evidence.
Critical strengths
- The event-level task has clear value, and the annotation work is thorough: the paper adopts multi-event annotation, provides an event-set detection task with practical research value, and constructs annotations containing event descriptions and temporal intervals through review by multiple annotators.
- Tests on several large models demonstrate the difficulty of the benchmark and its corresponding metrics.
Critical weaknesses
- The paper lacks comparative experiments with similar benchmarks. The incremental value of the new benchmark has not been sufficiently validated. The paper emphasizes event-set prediction and temporal localization, but compared with the closest work on physical implausibility detection and fine-grained evaluation, it does not sufficiently demonstrate what important diagnostic information these new designs provide that existing protocols cannot reveal.
- The benchmark available for testing is limited in scale, and its generated videos come from a single generator. This may limit fine-grained category analysis, and it remains unclear whether the results generalize to other generators and their different error types. The training and test data come from the same generator, and whether detection results and model rankings generalize to the error distributions of other generators has not been verified. The test set contains only five generated videos with no events, which is insufficient to adequately evaluate false positives and empty-list recognition on generated videos.
Questions for authors
Are there plans to release the complete dataset?
Could you supplement the evaluation with generated videos without physical defects and separately report results for real and generated videos without anomalies?
Thank you for asking us to clarify the dataset's contribution and diagnostic value.
W1: Dataset contribution. VIOLATE contains 556 generated videos and 2,708 human-reviewed claims: 1,552 confirmed violations and 1,156 incorrect claims. Alongside localized annotations, it retains rejected descriptions and intervals, including unsupported claims about videos containing other violations. Rejected claims test whether a model can reject a specific unsupported allegation even when the video contains genuine violations. The annotations therefore support both claim verification and open-ended detection. Sections 1–3 clarify this contribution and its relation to Spotlight and Physion-Eval.
W1: Annotation and diagnostic value. Reviewers add 318 violations missed by both proposal models (51.7% of evaluation references) and correct 81 intervals. Generated evaluation videos receive two independent reviews and adjudication. GPT-5.5 reaches 54.5% verification accuracy versus 51.8% always-reject, while Opus-4.8 recovers 34.8% of annotated violations in detection (Sections 3.2–3.3 and 5.3–5.6; Tables 2–4, 14, and 16).
W2: Evaluation correction. The original 170/155 event-containing/empty-target partition included three real videos from the 25 annotation controls among event-containing videos. Their auxiliary annotations are excluded from scoring by protocol, which restricts event references to generated videos and assigns empty references to real controls. The corrected partition is 167/158 on the same 325 videos: 167 generated videos with 615 event references, eight generated empty-target videos, and 150 real controls. The earlier count of five generated controls was an accounting error; no generated videos were added or relabeled (Section 3.3; Appendix E, Table 12).
W2: Generated controls. We report source-specific specificity: E4B/GPT-5.4 from-SFT scores 0.458 on eight generated controls and 0.813 on 150 real controls, versus 0.795 overall. Section 5.4 discusses the gap; Section 7 states the limited generated-control and single-generator coverage (Table 13).
Added experiments and audit. We added SFT-initialized final distillation and report detector results over three seeds. SFT uses 1:1 sampling of event-containing and empty-target videos (Sections 5.1–5.5; Table 5). A new 50-video human audit assesses equivalent matches (the same violation) and partial matches (a correct component or a broader description including the reference violation). Humans select 90.7% of automated equivalent pairs and 68.4% of partial pairs, with 96.2% grade agreement conditional on shared pairs (Section 3.3; Appendix H.5, Table 15).
Release. Upon publication, we will release videos, event descriptions and intervals, human verdicts, splits, prompts, configurations, and scoring code, including confirmed violations and rejected claims (Reproducibility Statement).
Proposal review (paper Table 2). Percentages are over each model's proposals.
The 297 accepted or retimed proposals plus 318 human-added events yield 615 evaluation references (Section 3.2; Appendix B).