- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces TA-Bench, a dataset for dense diagnosis of temporal artifacts in AI-generated videos. It represents an artifact with a temporally continuous severity field, constructed through automatic triage and mask generation followed by human verification and severity annotation. The paper also proposes Video Diffusion Diagnoser (VDD), which fine-tunes pretrained video diffusion models to predict this field in one step, and compares it with segmentation and video-understanding baselines on in-distribution and held-out-generator test sets.
Critical strengths
-The problem formulation is meaningful and timely. The paper identifies a genuine gap between per-frame visual realism and temporal consistency. Its distinction between global consistency, object permanence, and appearance constancy gives the task a useful structure.
-
The proposed annotation target is more expressive than a static object mask. A spatial mask weighted by time-varying severity can represent artifacts that appear only during part of an object's trajectory. The crop-consistency property is also practically relevant for temporal augmentation.
-
The paper makes a substantial data-construction effort. It reports 6,945 videos, 8,618 masks, a held-out-generator set, and human verification/refinement. The comparison across segmentation, self-supervised video, and generative backbones is a useful starting point.
Critical weaknesses
-
The annotation quality is not independently validated at the level required by the benchmark's main claim. Each initial candidate mask is assigned to one annotator; cross-annotator review is described for regenerated masks, not for accepted masks. The supplement reports Krippendorff's alpha = 0.74 for severity on a shared set of 50 videos, but it does not report agreement or an independent audit for (i) taxonomy assignment, (ii) artifact presence and temporal extent, or (iii) spatial mask accuracy. This is especially important because the initial triage, referring expression, and candidate mask all rely on models, while the proposed continuous target multiplies a binary mask by interpolated subjective severity. The pass/reject/refine percentages alone do not estimate label error. A blind multi-annotator audit with per-category mask/temporal agreement and a comparison against the final labels is needed.
-
The train/test protocol does not establish independence or support the claimed generalization. The ID test set is described only as a held-out 10% of videos. The paper does not state whether splits are grouped by original source video, prompt, source dataset, generator, or near-duplicate/derived clip. This matters because the main data originate from a small number of source datasets and the global artifacts are synthetically injected into selected source videos. A video-level random split can place highly related videos or synthesized variants in train and test. For the 200-video OOD set, the prompt distribution, artifact categories/severity, annotation procedure, and selection from the raw candidate pools are not reported, so it is difficult to tell whether it is a controlled generator shift or a different data distribution. Please provide the full split manifest or at least group-level counts, duplicate checks, and results under a source-video/prompt-disjoint split.
-
The empirical evidence does not isolate the claimed advantage of video diffusion priors. VDD-Wan2.2 is compared with VideoMAEv2 and V-JEPA using lightweight dense decoders, but the paper gives no matched-capacity, matched-optimization dense-prediction baseline or training variation. The strongest continuous-mask results on ID are instead obtained by VDD-ReferAnything, whose stated pretraining is video segmentation rather than video generation; it also leads most OOD metrics. Thus the current results do not cleanly separate the benefits of diffusion/generative pretraining, segmentation pretraining, backbone scale, and the proposed one-step objective. The claim that generative priors are the key reason for the gains should be narrowed or supported by controlled frozen-feature, from-scratch, and matched-backbone/pretraining comparisons, with multiple seeds and uncertainty estimates.
-
The broad claim that existing models are “temporally blind” rests on a weak and partly automated probe. Section 4.1 evaluates only 200 intermediate TA-Bench samples, maps text outputs to masks via ReferAnything, and uses an MLLM to judge semantic correspondence. Although the paper mentions a human-validated upper bound, it does not report the numerical human-vs-automatic agreement, sample selection, prompt protocol, or the actual aggregate outcomes clearly enough to quantify evaluator error. This chained evaluation can conflate a model's temporal reasoning with failures of the evaluator or referring-segmentation component. The conclusion should be limited to the tested protocol unless it is validated against a larger blind human study and direct task-specific baselines.
Questions for authors
-
Annotation reliability: For a blind, independently double-annotated subset, can the authors report agreement for artifact existence/category, temporal onset-offset, spatial masks (e.g., framewise IoU/boundary agreement), and severity? How many initially accepted masks, rather than only regenerated masks, were checked by a second annotator? Please also describe annotator expertise, instructions, compensation, and the adjudication procedure.
-
Split and OOD construction: Is the 10% ID split grouped by original video, prompt, source dataset, and generated/derived variant? Are clips or synthetic transformations of the same source video prevented from crossing splits? For the OOD set, report per-generator/category counts, raw-pool filtering/selection criteria, prompts, duration/resolution, severity distribution, and whether the annotators knew generator identity.
-
Causal source of VDD gains: Can the authors add matched experiments that disentangle video-diffusion pretraining, segmentation pretraining, parameter count, and the VDD objective? Please include multiple seeds or confidence intervals. Why does VDD-ReferAnything, rather than a video-generation-pretrained VDD, achieve the best ID continuous metrics and most OOD metrics?
-
Probe protocol: Please report the complete results of the 200-sample probe, the human-validated upper bound, and automatic-versus-human agreement. Can a blind human study directly evaluate whether each model identifies the correct temporal artifact, without routing text through another MLLM and ReferAnything?