acceptodds
Withdrawn from ICLR 2027

Benchmarks Lie, Signals Reverse: A Forensic Audit of AI-Generated Video Detection

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

AI-generated video detection has run an implicit measurement program for a decade: assemble real and generated video, train a separator, report benchmark accuracy. We audit the program and report its failure mode. Without precedent, to our knowledge, the field's strongest prior, motion amount, reverses sign across generator generations, fake-to-real flow ratio 0.362 on a 2024 generator against 3.342 on Seedance-2.5, and the 2026 wall is a detector-reference pairing: on Seedance every confounded-corpus system fails or inverts, and the standardized corpus's own system reads 0.4478 below chance on one reference family, above on another. The infrastructure failure is published, VidAudit's: of ten surveyed families four fall to zero-feature container rules that decode no frames, and on GenVidBench a duration threshold reaches a deployable train-fit 79.87 official while a constant FAKE prediction scores 80.00, above ten of eleven published systems. The surveyed signals fail our falsification battery: eight gates, each with a demonstrated kill, leave 0 of the 17 tested frozen-estimator families standing as a universal detector; the battery and family list co-evolved, so the zero reads as an exploratory falsification ledger, not a closed census. The systems layer survives only under a container-standardized protocol: 0.865 pooled AUC zero-shot on the one standardized benchmark, a hundred in-family labels reading 0.996 on the wall pool, near-duplicates unexcluded, off-family content ranking 0.744 with 8% detected. The limits are stated, not softened: single-seed rows, a small wild track, a frozen-estimator survey. The contribution is the protocol, the falsifications, and a label-priced adaptation ladder.

Primary area:
datasets and benchmarks
Keywords:
AI-generated video detection, deepfake detection, benchmark auditing, dataset contamination, shortcut learning, spurious correlations, generalization

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_39HgOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The manuscript audits AI-generated video detection benchmarks and their purported signals. It studies container, motion, duration, resolution, content, and reference-corpus effects, applies a multi-gate falsification battery, and reports that no tested signal family survives as a universal detector.

Critical strengths

The paper raises an important concern that benchmark performance can be driven by collection artifacts rather than generation. Its interventions are more informative than simply warning about dataset bias.

Critical weaknesses
  1. It feels exhausted to comprehend a 43-page manuscript without any illustration nor visualization.
  2. The largest conclusion is not as closed or decisive as its presentation suggests. The candidate family list, audit gates, and tested interventions co-evolved with the observations. Consequently, ''zero of 17 universal survivors'' is an exploratory result over a chosen audit battery, not evidence that universal detection signals do not exist.
  3. The manuscript also shifts between several different claims: benchmarks contain shortcuts, detectors fail under distribution shift, published rankings can reverse, and the field needs a new measurement protocol. These are related but not interchangeable. The self-falsification examples show that the authors' own earlier claims failed under their controls, but they do not validate the audit tool on unknown benchmarks or protect it from its own selection effects. The work is potentially useful as a cautionary case study, but it currently presents a local audit as a field-wide verdict.
Questions for authors
  1. Were the family list, gates, thresholds, and inclusion rules frozen before inspecting the results?
  2. How would the conclusions change under a newly constructed, generator-disjoint signal set selected by an independent group?
Reviewer_idMsOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents a forensic audit of AI-generated video (AIGV) detection benchmarks, focusing on dataset confounding, shortcut learning, and cross-generator generalization. Its main findings include: (i) container-level shortcuts substantially compromise existing benchmarks (ii) none of 17 tested signal families universally survives an eight-gate falsification protocol; (iii) motion statistics reverse direction across generator generations, and (iv) detection performance depends strongly on the real-video reference distribution. The paper further investigates container-standardized evaluation and few-shot adaptation.

Critical strengths
  1. The paper addresses the problem of shortcut signals in AIGV detection benchmarks: metadata-level discrepancies between real and generated videos (e.g., duration, frame rate) can systematically bias evaluation. This is a research topic that merits a systematic audit.
  2. The claimed audit experiments are broad in scope, examining existing methods from both the data side and the detector side, and several of the resulting conclusions are insightful.
Critical weaknesses
  1. As a submission to the Datasets & Benchmarks track, this paper only empirically re-confirms and re-states conclusions that prior work has already articulated (VidAudit, Cakiroglu et al. (2026) and Michels et al. (2026)), without proposing a new benchmark or proposing any solution for the problems it identifies. It therefore falls short of the novelty and contribution expected of an ICLR paper.
  2. The paper is poorly written. Both the main text and the appendix consist of dense, unstructured paragraphs and redundant result descriptions, while figures and tables illustrating the methodology and experimental results are largely absent. Many of the claimed 30 experiments are repetitive and add little evidential value.
  3. While the manuscript reports numerous quantitative findings, the main text contains only one quantitative results table and no experimental figures. Several central experiments such as the container interventions, the signal-family falsification, and few-shot adaptation lack consolidated result tables even in the appendix. This makes it unnecessarily difficult to verify experimental comparability, isolate the effects of individual controls, and assess whether the reported evidence supports the conclusions.
  4. Formatting violation: the body text on page 9 visibly overflows the template margins, which violates the ICLR submission format requirements.
  5. The appendix appears to be an unfiltered dump of the authors' entire working directory. It is extremely difficult to draw useful information and contains numerous irrelevant, apparently AI-generated documents.
Questions for authors

See weaknesses.

Reviewer_uhB6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper conducts a systematic forensic audit of benchmarks for AI-generated video detection and reports three core findings. (1) Real and generated videos are systematically separable at the container/metadata level (duration, frame rate, resolution, codec) in several benchmarks: zero-feature container rules that decode no frames separate real from fake (e.g., 99.83 oracle Mean Top-1; 98.96 accuracy on GenVideo-Val), a logistic regression on the duration scalar alone pools an AUC of 0.9968 against 0.7656 for the seventeen-feature bank, and on GenVidBench a constant FAKE prediction scores 80.00 on the matched protocol, above ten of eleven published systems. (2) The motion-amount prior reverses sign across generator generations: the fake-to-real flow-magnitude ratio reads 0.362 on CogVideoX but 3.342 on Seedance-2.5, and a 2024-trained motion expert lands below chance on Seedance. (3) On GenBuster, the one container-standardized benchmark, the physics backbone pools 0.865 AUC zero-shot on the mini split (single training draw), though the same system reads 0.771 on the id track, 0.702 on the wild track, and inverts at the Seedance wall. The authors conclude that scores measured on existing benchmarks cannot be interpreted as detection capability, prescribe protocol-level remedies (container columns beside accuracy, per-generation calibration), and offer a label-priced adaptation ladder (a hundred in-family labels reach 0.996 on the wall pool), which the paper itself does not present as a deployment procedure.

Critical strengths
  1. The audit targets a real problem, and the execution is disciplined. Old detection methods indeed struggle to keep up with current generation technology, and the reproducibility engineering is rare among audit studies: preregistration, a build that verified 1,820 claims with 0 failures and 0 warnings, and an eight-gate falsification battery (g1–g8).
  2. Several findings are concrete and independently checkable, with direct warning value for benchmark design: the duration-only learner's 0.9968 pooled AUC and the constant-FAKE 80.00 on GenVidBench are specific, recomputable results that any future benchmark protocol should be required to survive.
Critical weaknesses
  1. The core findings substantially overlap concurrent work, and the increment is narrower than the framing suggests. The container-shortcut phenomenon has already been documented by three 2026 audits (VidAudit, STALL, Michels et al.), and the paper itself credits VidAudit for the phenomenon ("The infrastructure failure is published, VidAudit's"). The paper's own increments — scoring container rules under the official metric, header-only execution, and tracing how a learned detector inherits the container field through its sampler — are real but incremental: the novelty lies in quantifying, not identifying, this confound. Combined with a density of coined vocabulary (gates, fences, wall, ladder) that the Appendix E glossary only partially rescues — the abstract's sentence defining "the 2026 wall is a detector-reference pairing" cannot be parsed without the §2 apparatus — the main text is hard to follow without the appendix.
  2. The headline rhetoric systematically outruns the paper's own qualifiers. The title asserts "Benchmarks Lie" and the abstract leads with failure language, while the same sentence constrains the central result to "an exploratory falsification ledger, not a closed census" and Table 1's "Not claimed" row disclaims universal detector failure. The "zero survivors" result comes from a ledger in which the eight gates and the 17 feature families co-evolved (g7/g8 were added after the rest of the battery had run), so the strongest quantitative claim rests on a post hoc protocol. The audit's evidential base is also thin in places the text does not emphasize: single-seed system rows, a small wild track, and reference AUCs that reuse the id track's real pool.
Questions for authors
  1. What is the precise increment over VidAudit, STALL, and Michels et al.? Please state it as a comparable experimental setup (which controls they ran, which you add), not as a terminological framework.
  2. The abstract's rhetoric ("Benchmarks Lie", "The infrastructure failure is published") is stronger than the paper's own qualifiers ("not a closed census"; Table 1's "Not claimed" row). Is this framing intentional?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.