acceptodds
Withdrawn from ICLR 2027

Achilles' Heel: Unveiling temporal artifacts in AI-Generated videos via video diffusion models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Despite rapid advances in visual quality, current video generation models still produce temporal artifacts, such as appearance drift or unexplained object emergence or disappearance that may be imperceptible from individual frames but obvious as time unfolds. Existing benchmarks predominantly lack attention to this type of artifacts and provide coarse-grained annotation. We propose Temporal Artifact Bench (TA-Bench), a benchmark dedicated to fine-grained diagnosis of temporal artifacts in AI-generated videos, comprising 9 generative models, 6,945 video with 8,618 continuous temporal masks and 214.3K frame-level annotations. To construct TA-Bench at scale, our pipeline combines automatic triage, mask generation, controllable synthesis, and 3 layer human verification. We evaluate various model families for temporal artifact localization, including general purpose multimodal language models (MLLMs), AIGC detection MLLMs and video segmentation models. Results show VLM-based methods depicting limited sensitivity to temporal artifacts, whereas generative video models achieve the strongest localization performance. Motivated by this finding, we propose Video Diffusion Diagnoser (VDD), a generation-for-detection approach that repurposes a pretrained video diffusion model to predict temporal continuous masks directly from video latent in a single feed-forward pass. Experiments on TA-Bench show that VDD outperform MLLM-based or segmentation-based models, suggesting that generative video priors provide a strong foundation for fine-grained temporal artifact perception.

Primary area:
datasets and benchmarks
Keywords:
AI-Generated Video, Benchmarks, Video diffusion model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_rVY8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces TA-Bench, a dataset for dense diagnosis of temporal artifacts in AI-generated videos. It represents an artifact with a temporally continuous severity field, constructed through automatic triage and mask generation followed by human verification and severity annotation. The paper also proposes Video Diffusion Diagnoser (VDD), which fine-tunes pretrained video diffusion models to predict this field in one step, and compares it with segmentation and video-understanding baselines on in-distribution and held-out-generator test sets.

Critical strengths

-The problem formulation is meaningful and timely. The paper identifies a genuine gap between per-frame visual realism and temporal consistency. Its distinction between global consistency, object permanence, and appearance constancy gives the task a useful structure.

  • The proposed annotation target is more expressive than a static object mask. A spatial mask weighted by time-varying severity can represent artifacts that appear only during part of an object's trajectory. The crop-consistency property is also practically relevant for temporal augmentation.

  • The paper makes a substantial data-construction effort. It reports 6,945 videos, 8,618 masks, a held-out-generator set, and human verification/refinement. The comparison across segmentation, self-supervised video, and generative backbones is a useful starting point.

Critical weaknesses
  • The annotation quality is not independently validated at the level required by the benchmark's main claim. Each initial candidate mask is assigned to one annotator; cross-annotator review is described for regenerated masks, not for accepted masks. The supplement reports Krippendorff's alpha = 0.74 for severity on a shared set of 50 videos, but it does not report agreement or an independent audit for (i) taxonomy assignment, (ii) artifact presence and temporal extent, or (iii) spatial mask accuracy. This is especially important because the initial triage, referring expression, and candidate mask all rely on models, while the proposed continuous target multiplies a binary mask by interpolated subjective severity. The pass/reject/refine percentages alone do not estimate label error. A blind multi-annotator audit with per-category mask/temporal agreement and a comparison against the final labels is needed.

  • The train/test protocol does not establish independence or support the claimed generalization. The ID test set is described only as a held-out 10% of videos. The paper does not state whether splits are grouped by original source video, prompt, source dataset, generator, or near-duplicate/derived clip. This matters because the main data originate from a small number of source datasets and the global artifacts are synthetically injected into selected source videos. A video-level random split can place highly related videos or synthesized variants in train and test. For the 200-video OOD set, the prompt distribution, artifact categories/severity, annotation procedure, and selection from the raw candidate pools are not reported, so it is difficult to tell whether it is a controlled generator shift or a different data distribution. Please provide the full split manifest or at least group-level counts, duplicate checks, and results under a source-video/prompt-disjoint split.

  • The empirical evidence does not isolate the claimed advantage of video diffusion priors. VDD-Wan2.2 is compared with VideoMAEv2 and V-JEPA using lightweight dense decoders, but the paper gives no matched-capacity, matched-optimization dense-prediction baseline or training variation. The strongest continuous-mask results on ID are instead obtained by VDD-ReferAnything, whose stated pretraining is video segmentation rather than video generation; it also leads most OOD metrics. Thus the current results do not cleanly separate the benefits of diffusion/generative pretraining, segmentation pretraining, backbone scale, and the proposed one-step objective. The claim that generative priors are the key reason for the gains should be narrowed or supported by controlled frozen-feature, from-scratch, and matched-backbone/pretraining comparisons, with multiple seeds and uncertainty estimates.

  • The broad claim that existing models are “temporally blind” rests on a weak and partly automated probe. Section 4.1 evaluates only 200 intermediate TA-Bench samples, maps text outputs to masks via ReferAnything, and uses an MLLM to judge semantic correspondence. Although the paper mentions a human-validated upper bound, it does not report the numerical human-vs-automatic agreement, sample selection, prompt protocol, or the actual aggregate outcomes clearly enough to quantify evaluator error. This chained evaluation can conflate a model's temporal reasoning with failures of the evaluator or referring-segmentation component. The conclusion should be limited to the tested protocol unless it is validated against a larger blind human study and direct task-specific baselines.

Questions for authors
  • Annotation reliability: For a blind, independently double-annotated subset, can the authors report agreement for artifact existence/category, temporal onset-offset, spatial masks (e.g., framewise IoU/boundary agreement), and severity? How many initially accepted masks, rather than only regenerated masks, were checked by a second annotator? Please also describe annotator expertise, instructions, compensation, and the adjudication procedure.

  • Split and OOD construction: Is the 10% ID split grouped by original video, prompt, source dataset, and generated/derived variant? Are clips or synthetic transformations of the same source video prevented from crossing splits? For the OOD set, report per-generator/category counts, raw-pool filtering/selection criteria, prompts, duration/resolution, severity distribution, and whether the annotators knew generator identity.

  • Causal source of VDD gains: Can the authors add matched experiments that disentangle video-diffusion pretraining, segmentation pretraining, parameter count, and the VDD objective? Please include multiple seeds or confidence intervals. Why does VDD-ReferAnything, rather than a video-generation-pretrained VDD, achieve the best ID continuous metrics and most OOD metrics?

  • Probe protocol: Please report the complete results of the 200-sample probe, the human-validated upper bound, and automatic-versus-human agreement. Can a blind human study directly evaluate whether each model identifies the correct temporal artifact, without routing text through another MLLM and ReferAnything?

Reviewer_XrqSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies temporal artifacts in AI-generated videos, which are often overlooked by benchmarks focused on frame-level visual quality. It introduces TA-Bench, with 6945 videos from nine generators and temporally continuous masks encoding artifact location, timing, and severity. The authors further propose Video Diffusion Diagnoser (VDD), which repurposes pretrained video diffusion models for dense artifact localization in a single feed-forward pass. Experiments show that VDD outperforms segmentation and video-representation baselines on both in-distribution and unseen-generator tests, demonstrating the usefulness of generative priors for fine-grained temporal artifact diagnosis.

Critical strengths

S1. The paper addresses a practically important problem by introducing a more fine-grained benchmark for evaluating modern video generation models.

One of the paper's main strengths is the practical motivation behind the benchmark design. Unlike prior benchmarks, TA-Bench provides dense annotations that jointly capture the spatial location, temporal occurrence, and continuous severity of artifacts. This allows the evaluation to go beyond simply determining whether an artifact is present and instead characterize where it appears, when it occurs, and how severe it is.

This level of supervision can support more precise evaluation of video generation models and may also facilitate follow-up research on localized and interpretable artifact diagnosis. The contribution of TA-Bench therefore extends beyond the proposed VDD model, as the benchmark itself could provide a useful resource for future work in this area.

Critical weaknesses

Since TA-Bench is a central contribution of the paper, the assessment mainly depends on whether the benchmark faithfully captures the failure modes that occur in real video generation models and whether the evaluation is designed rigorously enough to characterize those failures. From this perspective, the following two issues are the main factors affecting my recommendation.

W1. The paper does not establish whether the synthetic Global Consistency artifacts reflect failures that occur naturally in video generation models.

Because naturally occurring Global Consistency examples are scarce, TA-Bench augments them with synthetic exposure, white-balance, Gaussian-blur, Gaussian-noise, and saturation perturbations. This is a reasonable way to obtain controllable supervision, but it is not clear how well these handcrafted transformations match failures produced by current video generators. Similar visual symptoms could result from different underlying failure processes.

This matters because a model could partly learn the signatures of the synthetic operators rather than a more general notion of temporal inconsistency. A useful check would be to report performance on naturally occurring Global Consistency artifacts separately and compare training with real-only data against training with both real and synthetic examples. This would provide direct evidence for whether the synthetic augmentation transfers to genuine generation failures.

W2. The aggregate evaluation makes it difficult to understand what kinds of temporal artifacts VDD actually handles well.

TA-Bench separates temporal artifacts into Global Consistency, Object Permanence, and Appearance Constancy, but Tables 2 and 4 only report aggregate results, where these categories involve quite different failure modes. Global photometric fluctuation, object disappearance, and identity drift are unlikely to require exactly the same diagnostic capability. Category-wise results would therefore be important for understanding where VDD improves over the baselines and whether those improvements are consistent across artifact types.

The current metrics also provide limited information about temporal localization. MSE, CC, and SIM on the spatiotemporal masks penalize temporal errors, but they do not directly measure whether the predicted artifact begins and ends at the correct time. Since temporal diagnosis is central to the benchmark, an explicit metric such as temporal IoU or onset/offset error would complement the current evaluation.

Questions for authors

Q1. Could the authors report results separately for Global Consistency, Object Permanence, and Appearance Constancy? Also, have you considered a temporal-localization metric such as temporal IoU or onset/offset error in addition to MSE, CC, and SIM?

Q2. How well do the synthetic Global Consistency perturbations match naturally occurring failures from video generation models? Could you report performance on a real-only subset and an ablation with and without synthetic training examples?

Q3. Will TA-Bench be publicly released?

Reviewer_cg6vOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses temporal artifact diagnosis in AI-generated videos by introducing TA-Bench and a video diffusion-based method, VDD. TA-Bench provides 6,945 videos with fine-grained annotations of artifact locations, temporal occurrences, and severity, while VDD leverages pretrained video diffusion models to predict temporal artifact masks in a single forward pass. Experiments demonstrate that VDD outperforms several existing baselines on both in-distribution and out-of-distribution benchmarks. The main contribution lies in establishing a fine-grained temporal artifact benchmark and exploring generative priors for artifact diagnosis. However, I have several concerns as detailed in the Questions section.

Critical strengths
  • The paper addresses the important yet underexplored challenge of temporal artifacts in AI-generated videos.
  • TA-Bench provides fine-grained annotations of temporal artifacts, benefiting future research.
  • VDD achieves competitive performance on both ID and OOD test sets, demonstrating the potential of video generative priors.
Critical weaknesses

The paper has several important weaknesses. Please see the Questions section for details.

Questions for authors

Major Questions

  1. Although TA-Bench focuses on temporal artifacts, VDD adopts a generic video-to-mask regression framework without explicitly modeling artifact-specific temporal inconsistencies. How does the proposed method effectively exploit the characteristics of temporal artifacts beyond supervised mask prediction?

  2. The paper attributes VDD's effectiveness to the spatiotemporal priors of video diffusion models, but this claim lacks direct evidence. Could the authors provide single-frame and controlled temporal-order ablations to demonstrate that VDD genuinely captures temporal inconsistencies rather than relying on static visual artifacts?

  3. The proposed single-step transport objective appears mathematically equivalent to residual latent regression. What advantages does this formulation offer over direct mask prediction or conventional feature-based segmentation using the same pretrained backbone? Additionally, the authors should further clarify its novelty relative to existing diffusion-based anomaly detection methods.

  4. Although several existing methods are evaluated on TA-Bench, the experiments are limited to the newly proposed benchmark. Could the authors evaluate VDD on existing artifact benchmarks, such as BrokenVideos, and provide further comparisons with prior methods to demonstrate its generalization ability?

  5. TA-Bench primarily contains videos with annotated artifacts. How does VDD perform on artifact-free generated videos? Without evaluating false positives on clean videos, it is difficult to assess the reliability of the proposed diagnostic method.

Minor Questions

  1. TA-Bench is a major contribution, but its statistical analysis and visualizations remain limited. Could the authors provide more comprehensive analyses of artifact category distributions, temporal durations, severity distributions, and representative examples across different generators?

  2. VDD uniformly samples 73 frames per video, potentially missing short-lived artifacts. How sensitive is its performance to temporal sampling density, particularly for brief anomalies?

  3. For artifacts such as spontaneous object disappearance, how are spatial masks defined when the anomalous object is no longer visible? A clearer explanation of the annotation rules would strengthen the validity of the proposed benchmark.

  4. Several numerical results in the main text do not match the corresponding tables. For example, the Scaling Behaviour paragraph (Page 9) reports ID IoU improving from 0.1593 to 0.2688, while Table 2 shows 0.2111 to 0.2688. Could the authors verify and correct these discrepancies?

Reviewer_zc2nOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces temporal artifact diagnosis for AI-generated videos and constructs TA-Bench, which provides continuous masks annotating artifact location, timing, and severity. The authors also propose VDD, which repurposes pretrained video diffusion models to predict these masks, and evaluate it on both in-distribution videos and videos from unseen generators.

Critical strengths
  1. Practically meaningful task. The paper addresses temporal artifacts in AI-generated videos by identifying their location, timing, and severity, providing a valuable direction for fine-grained video quality diagnosis.
  2. Valuable benchmark contribution. TA-Bench covers multiple video generation models and combines semi-automated annotation with human verification, providing a foundation for training and evaluating temporal artifact diagnosis methods.
Critical weaknesses

1.Inconsistent experimental values between the text and the table. Lines 459–462 state that random temporal cropping increases IoU from 0.1593 to 0.2369, decreases MSE from 0.02260 to 0.02186, and increases CC from 0.2004 to 0.2144. However, the corresponding initial values in Table 3 are 0.2111, 0.02248, and 0.2072, respectively, which differ from those reported in the text.

2.Differences across evaluation metrics require clarification. I am not sufficiently familiar with the CC metric, but I notice that some methods show relatively limited differences in MSE and SIM, yet substantial differences in CC. Could the authors explain whether this behavior is reasonable?

3.Limited readability of some figures. In Figure 4, overlapping text boxes obscure the referring expressions, making them difficult to read in full.

Questions for authors

The initial IoU, MSE, and CC values reported in lines 459–462 differ from those in Table 3. Which values are correct, and would correcting them affect the reported magnitude of improvement?

Some methods show relatively limited differences in MSE and SIM but substantial differences in CC. Could the authors explain why this occurs and which aspects of the predictions these metrics capture?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.