acceptodds
Under review as a conference paper at ICLR 2027

Can AI-Generated Video Detectors Keep Up?

19 views

Abstract

Detecting AI-generated videos (AIGVs) has become increasingly important as powerful video generation models continue to emerge. However, existing benchmarks provide limited indications of how detectors will perform in practical deployment for two key reasons: (1) strong performance on internal test sets often fails to transfer to broader data distributions; and (2) commonly used one-to-many protocols and non-temporal splits do not conform to actual application. We introduce AIGV-Annual, a million-scale, temporally partitioned and extensible benchmark for AI-generated video detection, covering videos from 100 fake generators and 13 real sources. AIGV-Annual establishes a cumulative many-to-many protocol, in which detectors are trained on generators released before each temporal cutoff and evaluated on those released afterward, enabling the measurement of temporal decay and update gain under continual generator evolution. Using AIGV-Annual, we systematically evaluate 18 supervised baselines and 16 zero-shot configurations across multiple temporal cutoffs. Our experiments reveal substantial out-of-distribution degradation, temporal decay and varying update gains from newer training data. We further introduce a spatial-temporal detector. Extensive experiments demonstrate that our method outperforms state-of-the-art methods on existing benchmarks and in-the-wild data.

Primary area:
datasets and benchmarks
Keywords:
AI-generated video detection, benchmark

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_w2YuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper raises an interesting question: can AI-generated video detectors trained today reliably detect videos produced by the next generation of AI models? To address this question, the authors first establish the generalization limitations of existing detectors through evaluation on large-scale datasets. They then use these findings to guide the design of a new detector. Extensive experiments support the paper’s motivation, substantiate its claims, and evaluate the effectiveness of the proposed method.

Critical strengths

1 A key contribution of the paper is its extensive empirical support for an intuitive observation: detector performance varies substantially across groups of future generators, and many methods exhibit an overall decline over time. In light of these findings, the proposed benchmark has considerable practical value.

2 The proposed method is based on findings from extensive experiments, and the ablation studies further support its effectiveness.

Critical weaknesses

1 Generalization is a challenge across many research areas and is particularly pronounced in AI-generated video detection. Approaches such as online and continual learning have been developed to address this challenge. Could the authors clarify whether the proposed method offers a more effective solution than these approaches?

2 The paper’s methodological contribution appears somewhat limited, as the proposed method seems to be a straightforward combination of existing techniques. The paper might present a more focused research narrative by concentrating on introducing the benchmark and systematically analyzing the decline in detection performance on future generative models.

Questions for authors

1 Although the proposed method demonstrates effectiveness, is it a suitable approach to addressing generalization challenges, particularly compared with techniques such as continual learning?

2 Could the strong performance of the proposed method be primarily attributed to its use of a stronger backbone or other recent advances?

Reviewer_vZk4OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces AIGV-Annual, a roughly 1.29M-video benchmark organized by generator release periods, and a cumulative training/evaluation protocol intended to measure generalization to later generators. It evaluates 18 supervised baselines and 16 off-the-shelf configurations, and proposes a detector that combines a frozen, DDA-pretrained spatial branch with temporal patch attention and cross-attention fusion.

Critical strengths
  1. The emphasis on generator evolution is practically important, and evaluating multiple future cohorts provides more information than a single static split. The benchmark covers diverse real sources and generator versions, with detailed source statistics in the appendix. This breadth and the extensible temporal organization could make the benchmark useful for studying detector generalization.

  2. The reported detector improvements are substantial and supported by several complementary studies. Table 5 shows average balanced-accuracy gains of 6.2, 5.0, and 6.3 points over the strongest listed baseline across three training cutoffs. Tables 6–7 provide additional evaluations, while component, backbone, augmentation, and degradation studies help interpret the improvements. In particular, the DINOv2 results in Table 9 indicate that the gains are not explained solely by the newest backbone.

Critical weaknesses
  1. The deployment interpretation and evaluation boundary need to be specified more precisely. Section 4.1 restricts time partitioning to fake generators, whereas the main model uses DINOv3 (2025) and the fixed real-training pool includes UltraVideo dated June 2025 (Tables 9 and 15). A retrospective generator-held-out experiment is legitimate, but it does not by itself reproduce a detector available at each historical cutoff. The backbone, DDA/VAE pretraining, data provenance, validation split, and threshold/checkpoint selection are not sufficiently specified to establish that interpretation. This affects the scope and auditability of the central temporal-deployment claim; a cutoff-compatible control or a narrower claim would address it.

  2. The pooled results do not yet establish broad generator-level and cross-source generalization. Seedance-2.0 contributes 11,166 of 18,928 videos in 2026-H1, while Sora2, Veo 3.1, Vidu-Q2, and Marey together contribute only 340 of 30,437 videos in 2025-H2 (Tables 13–14). The BAcc definition balances real and fake classes but does not specify equal generator weighting, so cohort averages can conceal performance on small or difficult sources. In addition, Appendix B.1 lists AIGVDBench and GenVideo-100K among aggregated sources, while related benchmarks appear in external evaluations. Official training splits are a useful precaution, but a global overlap/near-duplicate audit and sample-level provenance manifest are not described. This is an unresolved independence risk, not evidence of actual leakage. Generator-macro and source-resolved results together with a documented overlap audit are important for the external-validity claim.

Questions for authors

Clarifications or checks that could materially change the assessment:

  1. Is the intended protocol generator-time-only or end-to-end time-faithful? Please identify the backbone/checkpoint, DDA/VAE pretraining, and relevant data provenance. Can the historical-deployment claim be supported by a cutoff-compatible control, or should it be narrowed to retrospective generator-held-out generalization?

  2. How is public release time defined consistently across paper announcements, model access, and availability of collected videos? Table 12 dates Sora to February 2024 and Table 13 dates STIV to September 2025; the exact versions and sample provenance matter for these assignments.

  3. Do generator-macro, complete per-generator detection rates, worst-generator results, and real-source false-positive rates support the pooled findings? Please include uncertainty estimates, especially for small generator groups.

  4. What sample-level overlap and near-duplicate checks were applied across aggregation sources, training pools, and external evaluations? Can a provenance manifest and a controlled preprocessing/source-bias check establish independence?

  5. Are all model, checkpoint, and threshold selections based on past-only validation without future-test labels? Please specify calibration, sampling, architecture/pretraining and baseline configurations, and report AUC and low-FPR operating points. A matched-data/compute control would help distinguish the effect of newer generators from a changed training budget in the update-gain analysis.

Reviewer_k5joOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on a critical pain point in AI-generated video detection—the oversight of generators' release chronological order—and innovatively constructs a million-scale, strictly time-partitioned benchmark, AIGV-Annual (comprising 100 fake generators and 13 real video sources), while proposing a detection architecture that integrates spatio-temporal features with cross-attention. This study establishes an important benchmark for scientifically evaluating detectors' generalization capabilities against future unseen models.

Critical strengths
  1. The paper keenly highlights the critical pain point in the current AI-generated video detection domain—namely, the oversight of generators' release chronological order—and innovatively proposes a large-scale benchmark, AIGV-Annual, strictly partitioned by time, laying a solid foundation for objectively evaluating detectors' generalization capabilities against future unseen models.

  2. The benchmark constructed in this study is unprecedentedly large in scale and highly diverse, encompassing 100 mainstream fake generators and 13 real video sources, thereby comprehensively covering various rapidly evolving video generation technologies in recent years.

  3. Beyond introducing a high-quality, large-scale dataset, the paper specifically designs a detector that integrates spatio-temporal features with cross-attention, demonstrating superior accuracy and practical value across multiple cross-temporal and cross-generator benchmark evaluations.

Critical weaknesses

1 The proposed detection method and benchmark primarily focus on fully synthesized videos, providing insufficient coverage for more challenging real-world content safety scenarios, such as localized manipulation, face swapping, and hybrid edits combining real and generated content.

2 The study lacks a systematic evaluation of the proposed complex spatio-temporal model regarding parameter count, memory footprint, computational complexity (FLOPs), and inference latency (FPS), making it difficult to fully assess its deployment overhead in large-scale streaming or real-time content moderation scenarios.

3 Ablation studies indicate that model performance heavily relies on the underlying pretrained vision foundation models (e.g., DINOv3), failing to sufficiently isolate and prove the independent gains of the designed architecture compared to simply adopting a stronger backbone.

4 Although the paper illustrates performance metric drops under different temporal cutoffs, it lacks an in-depth qualitative analysis of typical failure cases (Bad Cases) to diagnose which specific generator architectures, dynamic scenes, or forgery artifacts cause the model to fail.

5 Tables 2 and 3 in the paper use an excessively small font size, resulting in a crowded and barely readable layout that severely impairs the reading experience for key experimental data and critical performance comparisons.

Questions for authors

See the weekness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.