acceptodds
Under review as a conference paper at ICLR 2027

VIOLATE: Evaluating Video-Language Models on Fine-Grained Physical Implausibility

Abstract

Model-generated videos can look realistic while containing brief implausible events. Video-level evaluations cannot show which event occurred, when it occurred, or whether several events occur in the same video. We introduce VIOLATE, a benchmark for fine-grained physical implausibility detection. It contains 556 human-annotated synthetic videos with 1,552 events. Each annotation describes what goes wrong and when it happens. We also use 150 real videos to measure false detections. The benchmark supports two tasks. In event verification, a model decides whether a proposed implausible event is actually visible. In open-ended event detection, the model receives only the video and must find, describe, and localize every implausible event, or return no event when the video is plausible. Current models struggle with both tasks. GPT-5.5 reaches the best off-the-shelf Matthews correlation coefficient of for verification, while the best fine-tuned model reaches ; both values show weak discrimination. For open-ended detection, Claude Opus 4.8 reaches F1 for matching event descriptions and F1 for their time intervals. We propose three-stage training with teacher distillation, reinforcement learning (RL), and self-distillation. These stages first learn from teacher-generated reasoning, then improve event recovery, and finally learn from the RL model's own predictions together with videos that contain no annotated implausible event. Its best variants reach semantic F1 and temporal F1. We will release VIOLATE to support evaluation and training for physical implausibility in model-generated videos.

Primary area:
datasets and benchmarks
Keywords:
Video Implausibility Detection, LLM Judge, Human Annotations

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_BhA1OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper focuses on VLMs’ detection of physically implausible events in generated videos, including their number and temporal localization. It introduces VIOLATE, containing 556 synthetic videos and 1,552 human-reviewed event annotations, and uses 150 real videos to evaluate false positives. The benchmark supports two tasks: determining whether a given event description and time interval are valid, and discovering all anomalous events based solely on the video. The detector adopts three training stages: using human event lists to assist a teacher in generating supervision signals, reinforcement learning based on human reference events, and self-distillation combining the model’s own predictions with empty-event samples. The paper validates its contributions through comparisons across multiple models, experiments across training stages, an initialization ablation, a video-level filtering cascade, a verifier-reward comparison, and qualitative examples. The existing results support the challenging nature of the task and demonstrate the potential of the training pipeline, but evaluation reliability and the interpretation of no-event recognition capabilities require further evidence.

Critical strengths
  1. The event-level task has clear value, and the annotation work is thorough: the paper adopts multi-event annotation, provides an event-set detection task with practical research value, and constructs annotations containing event descriptions and temporal intervals through review by multiple annotators.
  2. Tests on several large models demonstrate the difficulty of the benchmark and its corresponding metrics.
Critical weaknesses
  1. The paper lacks comparative experiments with similar benchmarks. The incremental value of the new benchmark has not been sufficiently validated. The paper emphasizes event-set prediction and temporal localization, but compared with the closest work on physical implausibility detection and fine-grained evaluation, it does not sufficiently demonstrate what important diagnostic information these new designs provide that existing protocols cannot reveal.
  2. The benchmark available for testing is limited in scale, and its generated videos come from a single generator. This may limit fine-grained category analysis, and it remains unclear whether the results generalize to other generators and their different error types. The training and test data come from the same generator, and whether detection results and model rankings generalize to the error distributions of other generators has not been verified. The test set contains only five generated videos with no events, which is insufficient to adequately evaluate false positives and empty-list recognition on generated videos.
Questions for authors

Are there plans to release the complete dataset?

Could you supplement the evaluation with generated videos without physical defects and separately report results for real and generated videos without anomalies?

1 comment

Thank you for asking us to clarify the dataset's contribution and diagnostic value.

  • W1: Dataset contribution. VIOLATE contains 556 generated videos and 2,708 human-reviewed claims: 1,552 confirmed violations and 1,156 incorrect claims. Alongside localized annotations, it retains rejected descriptions and intervals, including unsupported claims about videos containing other violations. Rejected claims test whether a model can reject a specific unsupported allegation even when the video contains genuine violations. The annotations therefore support both claim verification and open-ended detection. Sections 1–3 clarify this contribution and its relation to Spotlight and Physion-Eval.

  • W1: Annotation and diagnostic value. Reviewers add 318 violations missed by both proposal models (51.7% of evaluation references) and correct 81 intervals. Generated evaluation videos receive two independent reviews and adjudication. GPT-5.5 reaches 54.5% verification accuracy versus 51.8% always-reject, while Opus-4.8 recovers 34.8% of annotated violations in detection (Sections 3.2–3.3 and 5.3–5.6; Tables 2–4, 14, and 16).

  • W2: Evaluation correction. The original 170/155 event-containing/empty-target partition included three real videos from the 25 annotation controls among event-containing videos. Their auxiliary annotations are excluded from scoring by protocol, which restricts event references to generated videos and assigns empty references to real controls. The corrected partition is 167/158 on the same 325 videos: 167 generated videos with 615 event references, eight generated empty-target videos, and 150 real controls. The earlier count of five generated controls was an accounting error; no generated videos were added or relabeled (Section 3.3; Appendix E, Table 12).

  • W2: Generated controls. We report source-specific specificity: E4B/GPT-5.4 from-SFT scores 0.458 on eight generated controls and 0.813 on 150 real controls, versus 0.795 overall. Section 5.4 discusses the gap; Section 7 states the limited generated-control and single-generator coverage (Table 13).

  • Added experiments and audit. We added SFT-initialized final distillation and report detector results over three seeds. SFT uses 1:1 sampling of event-containing and empty-target videos (Sections 5.1–5.5; Table 5). A new 50-video human audit assesses equivalent matches (the same violation) and partial matches (a correct component or a broader description including the reference violation). Humans select 90.7% of automated equivalent pairs and 68.4% of partial pairs, with 96.2% grade agreement conditional on shared pairs (Section 3.3; Appendix H.5, Table 15).

  • Release. Upon publication, we will release videos, event descriptions and intervals, human verdicts, splits, prompts, configurations, and scoring code, including confirmed violations and rejected claims (Reproducibility Statement).

Proposal review (paper Table 2). Percentages are over each model's proposals.

Proposal modelProposalsAccepted unchangedRejectedRetimed
GPT-5.4511168 (32.9%)291 (56.9%)52 (10.2%)
Gemini-3.1-Pro44748 (10.7%)370 (82.8%)29 (6.5%)

The 297 accepted or retimed proposals plus 318 human-added events yield 615 evaluation references (Section 3.2; Appendix B).

Reviewer_AEzuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

VIOLATE is a benchmark for fine-grained physical implausibility in videos. Unlike methods that assign a single plausibility score, it formulates the task as open-ended event-set prediction: models must discover all physical, causal, and spatiotemporal consistency anomalies, describe each in natural language, and localize its temporal interval, returning an empty set when no anomaly exists. The dataset contains human-annotated synthetic videos with localized anomalous events, plus real videos for false-positive testing. The paper studies event verification and open-ended event detection. Since existing VLMs perform poorly on verification, the main detector does not use a verifier as a reward model. Instead, it uses a staged pipeline: privileged teacher distillation, GRPO with human event annotations as references, and self-distillation with many empty-target samples to restore plausible-video recognition. RL improves event discovery and temporal localization but makes the model predict anomalies for nearly all plausible videos; self-distillation restores high specificity while retaining most RL gains in event recovery.

Critical strengths

The problem definition is of practical value. This paper extends the evaluation of video physical plausibility from a coarse-grained video-level score to open-ended event-set prediction. This setting offers significantly higher diagnostic value than a single video-level score, and is particularly suitable for analyzing where and how generated video models violate physical or spatiotemporal consistency.

Critical weaknesses
  1. The no-event evaluation suffers from a strong domain confound. Plausible-video recognition is seriously confounded by real-vs-synthetic domain: all 170 anomalous test videos are synthetic, while 150 of the 155 no-event videos are real and only 5 are synthetic; Stage I/III empty-target supervision is also mostly real. Thus, the current specificity gain does not sufficiently show that the model truly recognizes "physically plausible synthetic videos," since it may partly rely on real-vs-synthetic appearance/domain cues to decide whether to output an empty set.
  2. All manually annotated synthetic videos were generated by Veo 3.1 Fast, leaving unclear whether the observed failure modes and trained detector generalize across generators or instead learn Veo-specific artifacts. Moreover, both the Stage-II RL reward and the final semantic evaluation use GPT-5.5 for semantic matching between predicted and GT descriptions, yet the paper lacks sufficient independent human matching validation or alternative-judge robustness analysis, making it hard to determine how much of the final gain stems from genuine event understanding.
Questions for authors
  1. Currently, 150 of the 155 no-event test videos are real, while only 5 are synthetic no-event videos. Can the authors report specificity separately for the real and synthetic no-event subsets? Alternatively, can they construct a larger set of physically plausible synthetic evaluation videos drawn from the same Veo distribution as the positive videos? In that case, would the conclusions change?
  2. All synthetic benchmark videos currently come from Veo 3.1 Fast. We suggest that the authors experiment with other video generators to verify the cross-generator generalization of the detection results.
  3. A question about the paper’s motivation: compared with existing video-level benchmarks or benchmarks based on predefined physical questions, what irreplaceable diagnostic value does open-ended event-level detection offer when practically evaluating video generation models? Can the authors provide more direct empirical evidence to support the necessity of this task design?
1 comment

Thank you for highlighting source confounds and evaluation reliability.

  • W1 / Q1: Evaluation correction. The original 170/155 event-containing/empty-target partition included three real videos from the 25 annotation controls among event-containing videos. Their auxiliary annotations are excluded from scoring by protocol, which restricts event references to generated videos and assigns empty references to real controls. The corrected partition is 167/158 on the same 325 videos: 167 generated videos with 615 event references, eight generated empty-target videos, and 150 real controls. The earlier count of five generated controls was an accounting error; no generated videos were added or relabeled (Section 3.3; Appendix E, Table 12).

  • W1 / Q1: Source-specific results. We now report specificity and false predictions separately for generated and real controls (table below). The large source gap limits what pooled specificity establishes about plausible generated videos (Sections 5.4 and 7; Tables 13–14).

  • W2: Human matching audit. We added an audit on 50 distinct videos, with outputs from five foundation models, 186 references, and 106 predictions. Equivalent matches describe the same violation; partial matches describe a correct component or a broader description that includes the reference violation. Humans select 90.7% of automated equivalent pairs and 68.4% of partial pairs. Grade agreement is 96.2% among shared pairs, not overall matching accuracy. This audit assesses correspondence agreement for foundation-model outputs; the shared reward/evaluation judge limitation remains (Section 3.3; Section 7; Appendix H.5, Table 15).

  • W2 / Q2: Generator coverage. We clarify that the findings concern Veo-3.1-Fast, with disjoint training/evaluation captions across 60 actions. Cross-generator transfer remains outside the evaluated scope (Sections 3.2 and 7; Appendix A).

  • Q3: Diagnostic value. 94.6% of event-containing evaluation videos have multiple violations, averaging 3.68 per video. Positive-only RL reaches 0.534–0.609 sF1 but only 0.002–0.004 specificity, showing why recovery and false alarms need separate evaluation. Verification uses 615 correct and 661 incorrect claims, all from generated videos, scored directly against human labels; video source cannot separate these classes (Sections 3.2–3.3 and 5.3–5.4; Tables 3–5 and 8).

  • Added training experiments. Results now use three-seed means and include SFT-initialized final distillation. SFT samples event-containing and empty-target videos at 1:1 (Sections 5.1–5.5; Table 5).

Source-specific results (paper Table 13). Three-seed means; eight generated and 150 real controls. fp/S and fp/R count predictions per empty-target video.

DetectorOverall specificityGenerated specificityReal specificityfp/Sfp/R
E4B/GPT-5.4, from-SFT0.7950.4580.8130.880.32
E4B/Opus, post-RL0.8950.4170.9200.750.12

Human matching audit (paper Table 15). Appendix H.5 also reports 12 human-only and 10 automated-only pairs.

ComparisonCountAgreement
Human-selected / automated-selected pairs64 / 6252 pairs in common
Same grade among shared pairs50 / 5296.2%
Automated equivalent pairs also selected by humans39 / 4390.7%
Automated partial pairs also selected by humans13 / 1968.4%
Reviewer_CB2COpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces VIOLATE, a benchmark for identifying, describing, and temporally localizing physically implausible events in videos, while returning an empty list for videos with no such event. It contains 556 human-annotated synthetic videos with 1,552 events, supplemented with 150 real evaluation videos. The paper studies both verification of supplied event claims and open-ended event detection. Since the evaluated verifiers discriminate poorly between valid and invalid claims, the proposed detector uses a three-stage training procedure: privileged teacher distillation, reinforcement learning against human event annotations, and self-distillation with empty targets. The strongest self-distilled variant achieves 0.518 semantic F1, 0.405 temporal F1, and 0.745 specificity.

Critical strengths
  1. The benchmark addresses a useful evaluation gap. Requiring a variable-size set of event descriptions and temporal intervals provides more diagnostic information than a single video-level plausibility judgment. Separately evaluating no-event videos is particularly valuable: the results show that high event-detection F1 can coexist with severe false detections on plausible videos.

  2. The paper provides informative empirical observations and a substantial human annotation effort. Evaluation annotations undergo independent review and adjudication, and annotators both reject incorrect model proposals and add missed events. The experiments report weak verifier performance, the loss of no-event recognition after positive-only RL, and its partial recovery after the final training stage. The filter-then-RL and verifier-reward alternatives also help contextualize the proposed approach.

Critical weaknesses
  1. The necessity of the three-stage procedure, and especially the independent value of self-distillation, is not sufficiently established. The sequence has an intuitive motivation, but Stage III simultaneously restores many empty-target videos, replaces positive ground-truth targets with RL predictions, continues from an RL checkpoint, and changes checkpoint selection to account for specificity. The observed recovery in specificity therefore does not isolate the benefit of using self-generated targets. A crucial missing control is continued training from the same RL checkpoint using human positive annotations and the same empty examples, data ratio, budget, and selection criterion. The RL-teacher-distillation ablation starts from the untuned base model, so it removes both prior task-specific distillation and RL training rather than isolating the contribution of RL initialization. The preliminary mixed-data RL attempts are described qualitatively but do not establish that a simpler, adequately tuned alternative is inferior. A method overview figure showing the data and supervision changes would also make this reasoning easier to follow; Figure 1 currently illustrates the task rather than the training pipeline.

  2. The final gains are useful but do not yet establish a strong and broadly generalizable detector. For E4B/GPT-5.4, Stage III lowers semantic F1 from 0.585 to 0.518 and temporal F1 from 0.459 to 0.405, while recovering specificity from 0 to 0.745. This is a meaningful improvement in no-event recognition, but the final semantic recall is only 0.379 and 25.5% of no-event videos still receive at least one prediction. Moreover, the filter-then-RL alternative achieves higher semantic and temporal F1 (0.541 and 0.426) at almost identical specificity (0.742), so the case for the proposed approach depends partly on efficiency, which would benefit from a quantitative cost comparison. Generalization is also difficult to assess because all synthetic videos come from one generator, and 150 of the 155 no-event evaluation videos are real. This correlation between video source and event presence leaves open whether some of the recovered specificity reflects source discrimination rather than physical plausibility understanding.

Questions for authors
  1. From the same RL checkpoint, how does Stage III compare with continued training on the original human event lists plus the identical empty-target examples, under matched data ratios, training budgets, and checkpoint selection? Can you also initialize the existing Stage III target-training ablation from the Stage I checkpoint to isolate the value of RL initialization?

  2. What positive/empty sampling ratios and empty-reference reward definitions were tried in mixed-data RL, and what were the corresponding detection and specificity results? This clarification would help assess whether the three-stage schedule is necessary rather than a consequence of the chosen training mixture.

  3. Can you quantify the inference-cost or latency advantage over filter-then-RL, alongside the detection and specificity results? The GPT-5.4 cascade outperforms self-distillation on both event F1 metrics with similar specificity, making this efficiency comparison important to the method's practical contribution.

  4. What evidence rules out real-versus-generated source cues as a contributor to specificity? Please report results separately on real and synthetic no-event videos. More plausible synthetic examples and evaluation on another generator would be useful extensions for testing this concern.

Presentation suggestion: add a pipeline overview showing each stage's initialization, positive and empty-target data, teacher or target source, loss or reward, and checkpoint-selection criterion.

1 comment

Thank you for the suggestions on training controls and presentation.

  • W1 / Q1: SFT initialization. We added final distillation from the SFT checkpoint across all four configurations. This tests whether the final detector needs RL-trained weights or can learn from the RL teacher's predictions starting from SFT. It achieves higher mean semantic and temporal F1 than RL initialization in each configuration (tables below; paper Tables 5 and 13). The effect of self-generated targets remains unisolated. We removed the claim that self-distillation is necessary (Sections 5.5 and 7).

  • W1 / Q2: Training evidence. Detector results now report three-seed means. SFT uses 1:1 sampling of event-containing and empty-target videos. Post-RL distillation exceeds initial SFT on mean semantic F1, temporal F1, and specificity in all four configurations. We removed the necessity argument based on qualitative mixed-data RL attempts (Sections 4.3 and 5.1–5.6; Table 5).

  • W2 / Q3: Cascade comparison. For E4B/GPT-5.4, from-SFT has similar event-F1 means to filtering and higher specificity: 0.548/0.410/0.795 versus 0.538/0.407/0.736 (sF1/tF1/specificity). Filtering performs better in E2B/GPT-5.4 (0.557/0.438/0.734 versus 0.498/0.394/0.612). We therefore describe configuration-dependent tradeoffs. Distillation uses one student call; the cascade uses a teacher call plus a conditional student call. We make no measured latency or cost claim (Section 5.5; Appendix F).

  • W2: Detector performance. E4B/GPT-5.4 from-SFT reaches 0.548 sF1, versus 0.431 for the strongest foundation baseline, with semantic recall 0.437 (Sections 5.4–5.6; Tables 5, 13–14).

  • W2 / Q4: Evaluation correction. The original 170/155 event-containing/empty-target partition included three real videos from the 25 annotation controls among event-containing videos. Their auxiliary annotations are excluded from scoring by protocol, which restricts event references to generated videos and assigns empty references to real controls. The corrected partition is 167/158 on the same 325 videos: 167 generated videos with 615 event references, eight generated empty-target videos, and 150 real controls. The earlier count of five generated controls was an accounting error; no generated videos were added or relabeled (Section 3.3; Appendix E, Table 12).

  • W2 / Q4: Source-specific false alarms. E4B/GPT-5.4 from-SFT scores 0.458 generated specificity and 0.813 real specificity, versus 0.795 overall. E4B/Opus post-RL scores 0.417/0.920, versus 0.895 overall. Section 5.4 discusses this gap; the eight generated controls provide limited evidence about plausible generated-video recognition (Section 7; Tables 13–14).

  • Presentation. Section 4.3 now has two parts: constructing the teacher through SFT and RL, then distilling its predictions with empty targets.

E4B/GPT-5.4 results (paper Tables 5 and 13). Three-seed means; event metrics use 167 generated videos and specificity uses 158 empty-target videos.

ConfigurationSemantic F1Temporal F1Specificity
Initial SFT0.3470.2350.597
Reference-based RL0.5800.4380.004
Final distillation from base0.4570.3310.897
Final distillation from SFT0.5480.4100.795
Final distillation post-RL0.4810.3570.785
Filter → RL0.5380.4070.736

Initialization comparison (paper Table 5). Cells report semantic F1 / temporal F1 / specificity.

Student / foundation teacherFinal distillation post-RLFinal distillation from SFT
E2B / Opus-4.80.465 / 0.351 / 0.7000.478 / 0.377 / 0.684
E2B / GPT-5.40.486 / 0.379 / 0.6540.498 / 0.394 / 0.612
E4B / Opus-4.80.489 / 0.372 / 0.8950.515 / 0.402 / 0.816
E4B / GPT-5.40.481 / 0.357 / 0.7850.548 / 0.410 / 0.795

Comments on the paper

1 comment

We thank the reviewers for their helpful feedback. We revised the paper to emphasize VIOLATE's dataset contribution: human-confirmed violations and rejected claims supporting event verification and open-ended detection. The training study examines how these annotations support event recovery, temporal localization, and fewer false alarms.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.