- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The manuscript audits AI-generated video detection benchmarks and their purported signals. It studies container, motion, duration, resolution, content, and reference-corpus effects, applies a multi-gate falsification battery, and reports that no tested signal family survives as a universal detector.
Critical strengths
The paper raises an important concern that benchmark performance can be driven by collection artifacts rather than generation. Its interventions are more informative than simply warning about dataset bias.
Critical weaknesses
- It feels exhausted to comprehend a 43-page manuscript without any illustration nor visualization.
- The largest conclusion is not as closed or decisive as its presentation suggests. The candidate family list, audit gates, and tested interventions co-evolved with the observations. Consequently, ''zero of 17 universal survivors'' is an exploratory result over a chosen audit battery, not evidence that universal detection signals do not exist.
- The manuscript also shifts between several different claims: benchmarks contain shortcuts, detectors fail under distribution shift, published rankings can reverse, and the field needs a new measurement protocol. These are related but not interchangeable. The self-falsification examples show that the authors' own earlier claims failed under their controls, but they do not validate the audit tool on unknown benchmarks or protect it from its own selection effects. The work is potentially useful as a cautionary case study, but it currently presents a local audit as a field-wide verdict.
Questions for authors
- Were the family list, gates, thresholds, and inclusion rules frozen before inspecting the results?
- How would the conclusions change under a newly constructed, generator-disjoint signal set selected by an independent group?