acceptodds
Under review as a conference paper at ICLR 2027

A Living In-the-Wild Benchmark for Measuring the Static-Benchmark Gap in AI-Generated Video Detection

Abstract

Video detectors that succeed on laboratory benchmarks can fail on videos shared online, where generators, editing, and platform processing change over time. We introduce VidTide, a benchmark that tracks this changing population through successive, fixed releases. Its April 2026 slice contains 21,504 clips with provenance-based labels; a blind audit finds 96.4% agreement and 0.6% clear label errors. Five detector families score 52–71% AUROC on a balanced evaluation subset, compared with their published 93–96% on GenVideo. Standard backbones trained on VidTide reach 90.5% mean AUROC with linear probes and 91.7% when selecting the better of probing and fine-tuning. Controlled evaluations examine compression, metadata shortcuts, and newer multimodal models; temporal and cross-dataset tests assess transfer. We release versioned manifests and download scripts, and measure URL availability to document reconstruction limits. The results support updating training-data coverage while preserving fixed test sets. Code and data: https://anonymous.4open.science/r/vidtide-anon-C0B8/

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
AI-generated video detection, living benchmark, video forensics, deepfake detection, in-the-wild evaluation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Nh2rOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces VidTide, a "living" in-the-wild benchmark for AI-generated video detection, built for repeated evaluation under a shifting content population. The April 2026 release (M0) contains 21,504 clips (11,502 AI-generated, 10,002 real) with provenance-based labels: positive labels rely mainly on platform AI disclosures (in China driven by mandatory labeling regulation) and on curated AI channels/subreddits abroad, while negative labels rest on the absence of such evidence, validated by a 200-clip Bilibili audit. Published GenVideo results of 93–96% AUROC drop to 52–71% zero-shot on the benchmark's balanced evaluation pool. The paper probes candidate explanations with controlled experiments, each with explicit scope caveats: a bitrate ladder over 8 re-encodes costs VIDTIDE-trained backbones only 1.73pp; on 486 matched real/generated pairs the metadata-only classifier falls to 55.03% AUROC while five visual probes hold at 88.29% (a 33.26pp matched advantage); and zero-shot MLLM performance rises across successive Qwen-VL detector generations (62.43% → 74.34%). Reverse transfer (linear probes trained on VidTide, evaluated on GenVideo-Val) reaches 67.8–77.8% versus 89.5% on M0-TEST.

Critical strengths
  1. The problem is real and important, and the response is well designed. The gap between static laboratory benchmarks and the in-the-wild distribution (new generator generations, platform transcoding, duration and resolution differences) is the central pain point of current deepfake detection, and the living-benchmark release mechanism is a reasonable answer to it. The paper honestly quantifies its own infrastructural decay (URL survival), which most benchmark papers do not do.
  2. The controlled probes are careful and honestly scoped. The bitrate ladder, the metadata matching on 486 pairs, and the detector-generation comparison each use paired bootstrap intervals and fixed content where possible, and the paper do not claim a full decomposition of the 31pp gap.
Critical weaknesses
  1. The label foundation of the negative (real) pool is thinner than the headline numbers suggest. Essentially all 10,002 real clips are Tier 2 absence-of-tag evidence, dominated by 9,336 unlabeled Bilibili uploads. The audit of this assumption is small and narrow: a 200-clip, Bilibili-only, non-stratified manual audit whose 95% CI upper bound (5.7%) is about 2.3x the 2.5% "likely AI" estimate. The cleanlab ensemble pass over 13,835 clips cannot separate detector error from label error (93 of its 100 most suspicious examples were real-labeled clips), and no watermark/C2PA or frequency-based screening of the real pool is reported. Moreover, the contamination bound (Eq. 2) assumes contamination independent of detector scores; structured contamination — undisclosed fakes that are hard to detect, precisely the current-generation ones — would violate it, and the manual audit cannot test this because its reference standard is again human visual judgment.
  2. Collection- and reproducibility-related biases are not fully quantified. Platform takedowns and deletions are plausibly correlated with labels, so comparisons across successive releases could shift sample difficulty in either direction; the paper fixes slices and marks retired rows, but does not quantify the effect. The private leaderboard subset limits independent reproduction of leaderboard scores. Finally, the binary labeling scheme ("entirely or predominantly by an AI model") neither covers nor flags spliced or partially generated content, and 74.1% of positive clips carry no generator tag, limiting per-generator attribution.
Questions for authors
  1. The paper plans to extend the manual audit to 500 stratified Bilibili unlabeled real clips. Can this extension add tool-assisted screening (watermark/C2PA inspection, frequency-artifact checks, detector ensembles) as a cross-check, and report the tool-human disagreement rate? If contamination is heterogeneous across strata, how sensitive are the leaderboard rankings?
  2. Can the crawl logs from each release (deletion/takedown events and retired rows, not only surviving clips) be used to estimate the direction and magnitude of the deletion bias across releases?
  3. For spliced/partially generated content, can the labeling protocol admit a third category? Even without relabeling, can the share of such content in the real and fake pools be estimated?
  4. The per-generator-family analysis (Appendix E.9) relies on a supervised family classifier with 46.7% out-of-fold top-1 accuracy (5 classes). What is the sensitivity of the per-generator conclusions to this classifier's error rate?
Reviewer_9q9qOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses the gap between laboratory evaluation and the detection of AI-generated videos circulated on online platforms. It introduces VIDTIDE, which combines recurring collection with frozen evaluation releases, provenance-based label evidence, and retrieval records. On its April 2026 release, the paper reports substantial degradation of existing detectors relative to their published laboratory results, improved performance after adaptation to VIDTIDE, and controlled evaluations examining compression, metadata shortcuts, and newer multimodal models.

Critical strengths
  • The benchmark targets an important evaluation gap. Sections 1-2 motivate evaluation on videos affected by changing generators, editing, and platform processing. Table 5 provides concrete evidence that strong published laboratory performance does not necessarily transfer to the collected platform videos.

  • The design separates ongoing collection from fixed evaluation releases. Section 3 proposes preserving historical slices while adding new ones, allowing methods to be compared on identical examples within a release. Section 3.3 also explicitly addresses URL availability and retrieval snapshots, which are practical concerns for a platform-sourced benchmark.

  • The paper includes independent label checks. Section 4 and Table 4 report blinded review by three annotators, distinguish agreement from clear label errors, and separately examine undisclosed generation among untagged videos. These checks acknowledge and examine the limitations of platform-derived labels.

  • The evaluation includes useful diagnostic controls. Section 5.4 examines additional compression and measured metadata shortcuts, while Table 7 compares specialist detectors and multimodal models on the same 1,000 clips. These experiments help assess alternative explanations for the reported performance.

Critical weaknesses
  • W1: The central claim of a continuously updated benchmark is insufficiently demonstrated. Figure 2 identifies M0 as April 2026 and marks the May and June releases as "upcoming," whereas the temporal evaluation in Section 5.3 contains only 99 clips. At the time of this review in October 2026, the main text does not establish whether subsequent frozen releases have actually been published. This does not demonstrate that updates have stopped, but it leaves the benchmark's distinguishing capability - repeated evaluation under an evolving distribution - largely unvalidated. A small temporal holdout provides preliminary evidence, but cannot establish the operation or value of the proposed monthly evaluation process.

  • W2: The source composition limits the interpretation and generality of the benchmark's scores. Table 2 shows that Bilibili contributes 79.8% of the release, while Reddit, YouTube, and official galleries contribute generated videos only. Consequently, the aggregate evaluation reflects a specific sampling distribution rather than broad coverage of online video. The Bilibili-only analysis in Section 5.3 is helpful, but does not rule out differences in subject matter, subtitles, or editing style between real and generated content within that platform. Moreover, Table 8 compares each platform's generated videos against the full real pool, so these scores cannot isolate platform-specific difficulty. Broader sourcing, including real and generated videos from the same additional platforms, would strengthen the benchmark's external validity.

  • W3: The transition from the motivating problem to the benchmark design could be clearer. Sections 1-2 motivate VIDTIDE through laboratory-to-platform performance degradation, but Sections 3-4 move directly into the construction pipeline without first giving readers a concise account of the suspected causes and corresponding design requirements. Section 5.4 provides relevant diagnostic experiments, but their connection to the preceding construction choices is not made sufficiently explicit. This is primarily an issue of argument structure: a short transition between Sections 2 and 3, outlining the evaluation questions and explaining how the design and later controls address them, would make the rationale for the benchmark easier to assess.

  • W4: The writing and organization lack a clear, coherent narrative. Beyond the motivation-to-design transition, the paper presents many construction steps and experiments without consistently explaining how they fit together and support the central claims. As a result, parts of the paper read as a list of procedures and results. Sections 3.2 and 5 introduce different data pools and experimental subsets, while key settings are scattered across table captions and footnotes. Readers must repeatedly cross-reference these descriptions to understand the evaluation setup and determine which results are directly comparable. Figure 2 emphasizes processing steps rather than the resulting data composition, and the legend and annotation layout in Figure 5 further reduces readability. Explaining experimental settings together in the main text and adding concise plots of platform-by-label composition, known versus unknown generator identity, and real versus generated video duration would make the benchmark easier to understand and assess. These presentation issues are repairable, but currently obscure the specification and interpretation of the evaluation.

Questions for authors
  1. Have M1 and M2, or any later frozen releases, been published? Please provide their release dates, version identifiers, sizes, and evaluations of fixed detectors across releases. If they have not been published, what explains the discrepancy with the monthly release schedule in Figure 2?

  2. How should the benchmark's intended scope be understood given its source composition? Table 2 shows that 79.8% of the release comes from Bilibili, while the other sources contain generated videos only. Is VIDTIDE primarily intended to evaluate detection on Bilibili, or to support broader conclusions about online video? What evidence supports the broader interpretation, and are there concrete plans to collect both real and generated videos from additional platforms? Such plans would be useful, although evidence from broader sourcing would be needed to establish broader validity.

Reviewer_BDjnOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces VIDTIDE, a continuously updated benchmark for detecting AI-generated videos in real-world web environments, designed to address the phenomenon where static, laboratory-based benchmarks become obsolete over time.

Critical strengths
  • Introduces the version-frozen rolling-slice dynamic paradigm to the task of AI-generated video detection, enabling effective tracking of detection performance degradation as generative models evolve.
Critical weaknesses
  • The benchmark comparison is incomplete, failing to address the latest benchmarks from 2026, such as MintVid (ICML 2026) and AIGVDBench (CVPR 2026).

  • Furthermore, the data sources are imbalanced, as the authors acknowledge in the Limitations section, nearly 80% of the data originates from Bilibili, and the ratio of authentic to synthetic content is also uneven.

Questions for authors
  • Fig. 1 specifies a range of 1k–4k px, whereas Sec. 3.1 mentions filtering and re-encoding to 720p.
  • The manuscript claims that fine-tuning on VIDTIDE can achieve an average AUC of over 90%, but how does the model perform in terms of generalization when evaluated on an equivalent dataset of authentic and forged samples from sources other than Bilibili?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.