- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces VidTide, a "living" in-the-wild benchmark for AI-generated video detection, built for repeated evaluation under a shifting content population. The April 2026 release (M0) contains 21,504 clips (11,502 AI-generated, 10,002 real) with provenance-based labels: positive labels rely mainly on platform AI disclosures (in China driven by mandatory labeling regulation) and on curated AI channels/subreddits abroad, while negative labels rest on the absence of such evidence, validated by a 200-clip Bilibili audit. Published GenVideo results of 93–96% AUROC drop to 52–71% zero-shot on the benchmark's balanced evaluation pool. The paper probes candidate explanations with controlled experiments, each with explicit scope caveats: a bitrate ladder over 8 re-encodes costs VIDTIDE-trained backbones only 1.73pp; on 486 matched real/generated pairs the metadata-only classifier falls to 55.03% AUROC while five visual probes hold at 88.29% (a 33.26pp matched advantage); and zero-shot MLLM performance rises across successive Qwen-VL detector generations (62.43% → 74.34%). Reverse transfer (linear probes trained on VidTide, evaluated on GenVideo-Val) reaches 67.8–77.8% versus 89.5% on M0-TEST.
Critical strengths
- The problem is real and important, and the response is well designed. The gap between static laboratory benchmarks and the in-the-wild distribution (new generator generations, platform transcoding, duration and resolution differences) is the central pain point of current deepfake detection, and the living-benchmark release mechanism is a reasonable answer to it. The paper honestly quantifies its own infrastructural decay (URL survival), which most benchmark papers do not do.
- The controlled probes are careful and honestly scoped. The bitrate ladder, the metadata matching on 486 pairs, and the detector-generation comparison each use paired bootstrap intervals and fixed content where possible, and the paper do not claim a full decomposition of the 31pp gap.
Critical weaknesses
- The label foundation of the negative (real) pool is thinner than the headline numbers suggest. Essentially all 10,002 real clips are Tier 2 absence-of-tag evidence, dominated by 9,336 unlabeled Bilibili uploads. The audit of this assumption is small and narrow: a 200-clip, Bilibili-only, non-stratified manual audit whose 95% CI upper bound (5.7%) is about 2.3x the 2.5% "likely AI" estimate. The cleanlab ensemble pass over 13,835 clips cannot separate detector error from label error (93 of its 100 most suspicious examples were real-labeled clips), and no watermark/C2PA or frequency-based screening of the real pool is reported. Moreover, the contamination bound (Eq. 2) assumes contamination independent of detector scores; structured contamination — undisclosed fakes that are hard to detect, precisely the current-generation ones — would violate it, and the manual audit cannot test this because its reference standard is again human visual judgment.
- Collection- and reproducibility-related biases are not fully quantified. Platform takedowns and deletions are plausibly correlated with labels, so comparisons across successive releases could shift sample difficulty in either direction; the paper fixes slices and marks retired rows, but does not quantify the effect. The private leaderboard subset limits independent reproduction of leaderboard scores. Finally, the binary labeling scheme ("entirely or predominantly by an AI model") neither covers nor flags spliced or partially generated content, and 74.1% of positive clips carry no generator tag, limiting per-generator attribution.
Questions for authors
- The paper plans to extend the manual audit to 500 stratified Bilibili unlabeled real clips. Can this extension add tool-assisted screening (watermark/C2PA inspection, frequency-artifact checks, detector ensembles) as a cross-check, and report the tool-human disagreement rate? If contamination is heterogeneous across strata, how sensitive are the leaderboard rankings?
- Can the crawl logs from each release (deletion/takedown events and retired rows, not only surviving clips) be used to estimate the direction and magnitude of the deletion bias across releases?
- For spliced/partially generated content, can the labeling protocol admit a third category? Even without relabeling, can the share of such content in the real and fake pools be estimated?
- The per-generator-family analysis (Appendix E.9) relies on a supervised family classifier with 46.7% out-of-fold top-1 accuracy (5 classes). What is the sensitivity of the per-generator conclusions to this classifier's error rate?