acceptodds
Under review as a conference paper at ICLR 2027

Dataset Biases and Shortcut Learning in Motion-Based AI-Generated Video Detection

Abstract

The visual quality of AI-generated videos has improved drastically in recent years, making it increasingly difficult for humans to distinguish between real and synthetic media. In this work, we evaluate the robustness and applicability of four state-of-the-art motion-based AI-generated video detectors. We identify significant preprocessing and sampling biases in three of the four methods and demonstrate that they account for a substantial portion of their reported performance. Furthermore, we find that these detectors are highly sensitive to motion patterns specific to their evaluation datasets, where AI-generated videos generally exhibit less inter-frame movement than real videos. We show that for all detectors, performance collapses to near-random levels when evaluated on a dataset that does not contain this motion bias. Additionally, through dataset rebalancing and the application of simple spatial augmentations, we observe severe performance degradation across all evaluated models. In contrast, we find that an existing frequency-based detector maintains strong performance across all evaluated datasets, suggesting that frequency-based approaches may offer a more generalizable path forward for AI-generated video detection. We hope that our work raises awareness towards these vulnerabilities and encourages the development of more representative, unbiased datasets and more robust evaluation protocols.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI-generated video detection, dataset biases, shortcut learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qng7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper highlights potential biases in existing AI-generated video detection datasets that may encourage detectors to learn shortcuts. The authors show experimentally that some detectors are highly sensitive to particular motion patterns in the evaluation dataset and perform poorly on a dataset without this motion bias. They also evaluate WaveRep, a frequency-domain baseline, and find that it substantially outperforms motion-based detectors.

Critical strengths

The paper addresses an important research question by directly examining shortcut learning in existing methods. Its focus on detector generalization has practical value for assessing the reliability of AI-generated content detection.

Critical weaknesses

1 Some details of the experimental design need improvement. In Table 6, AI-generated and real videos are filtered at different rates, which may introduce a distributional bias and complicate the comparison. Filtering both classes at the same rate would provide a better controlled evaluation.

2 The paper’s incremental contribution appears limited. Shortcut learning is already a recognized challenge in AI-generated content detection [1] [2]. Although the paper seeks to determine how much detection performance stems from dataset and preprocessing biases, its central contribution may amount to experimentally confirming a phenomenon that is already well known.

[1] Fang, Ziyuan, et al. "Real Data Lies: Unveiling and Closing the Quality Shortcut in Generalizable AI-Generated Video Detection." Forty-third International Conference on Machine Learning. 2026.

[2] Smeu, Stefan, et al. "Circumventing shortcuts in audio-visual deepfake detection datasets with unsupervised learning." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025.

Questions for authors

1 Could the authors clarify the paper’s contribution, particularly how it differs from prior work that has identified shortcut learning in AI-generated video detection?

2 The authors find substantial sampling and preprocessing biases in three of the four evaluated detectors. Could the authors explain why similar biases were not observed for Over-Coherence?

Reviewer_oRoZOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper systematically evaluates the robustness and generalizability of four motion-based AI-generated video detectors, revealing significant preprocessing and sampling biases. The results show that these detectors heavily rely on dataset-specific motion patterns. A frequency-based detection model demonstrates stronger robustness across datasets, suggesting a more promising direction for generalizable AI-generated video detection.

Critical strengths
  1. The paper performs a crucial reality check on recent motion-based AI video detectors. While these models claim near-perfect performance, the authors successfully reveal that a substantial portion of this accuracy stems from exploiting sampling and preprocessing biases, rather than actually detecting motion-related differences between AI and real videos.

  2. The authors systematically audit five recent evaluation datasets and uncover a major flaw: an imbalance in motion patterns. They show that in most datasets, AI-generated videos are heavily biased toward limited inter-frame movement, whereas real videos contain dynamic motion. This highlights that current benchmark datasets fail to accurately represent the rapidly evolving capabilities of modern AI video generators.

  3. By re-evaluating the detectors on temporally balanced datasets, the paper shows a significant drop in performance. Furthermore, it exposes a critical vulnerability. This proves that these models are relying on superficial shortcuts rather than robust features.

Critical weaknesses
  1. Insufficient Evidence to Support the Conclusions. The conclusions are not sufficiently supported by rigorous analysis and experimental evidence. Evaluating only four motion-based detectors and one non-motion-based detector is insufficient to conclude that the identified biases are specific to motion-based approaches.
  2. Unclear Definition of Biases. The paper lacks a clear and comprehensive explanation of the biases identified in the evaluated models and datasets. In particular, it remains unclear whether the observed performance degradation primarily originates from inherent limitations of the detection models, biases introduced during dataset construction, or flaws in the evaluation protocols. A more systematic analysis is needed to distinguish these factors.
  3. Limited and Unclear Contributions. The paper mainly identifies potential weaknesses in existing motion-based detectors and their evaluation frameworks, but provides limited investigation into how these issues could be addressed. Without exploring potential mitigation strategies or proposing a more reliable evaluation framework, the practical significance and novelty of the contributions remain unclear.
Questions for authors
  1. Beyond WaveRep, have the authors evaluated additional non-motion-based detection methods to further validate whether the identified biases are specific to motion-based approaches?
  2. Could the authors provide a more detailed explanation of the two identified biases, particularly their underlying causes and whether they originate from dataset construction, model design, or evaluation protocols?
  3. How should future benchmarks be constructed or detection models be designed to effectively eliminate or mitigate the identified biases and ensure a more reliable evaluation of genuine detection capabilities?
Reviewer_aFupOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the issues of dataset biases and shortcut learning in motion-based AI-generated video detection methods. By systematically testing four cutting-edge motion-based detectors, the authors find that these methods often rely heavily on low-level preprocessing biases, sampling biases, and the inherent motion pattern imbalances within datasets (e.g., AI-generated videos typically exhibit less inter-frame motion than real videos). The study demonstrates that when these biases are removed or evaluated on temporally balanced benchmarks, the detection performance of the models drops significantly. In contrast, frequency-domain-based methods exhibit better robustness and generalization capabilities. This work aims to point out flaws in current evaluations and promote the establishment of fairer, unbiased datasets and evaluation protocols.

Critical strengths
  1. The paper points out common dataset biases and shortcut learning issues in the current field of AI-generated video detection, revealing through in-depth analysis how external factors influence the high accuracy of some motion-based detectors, holding high reference value.

  2. The research design is relatively comprehensive, not only reproducing and testing four mainstream detection models, but also multi-dimensionally validating the practical impact of motion bias on model generalization through cross-dataset rebalancing, introducing spatial data augmentations, and comparing frequency-domain detectors.

3.The paper proposes specific improvement suggestions for data construction and evaluation protocols, providing a useful reference direction for future multimedia forensics research.

Critical weaknesses
  1. Although the paper points out the reliance of existing motion-based detectors on motion bias and sampling preprocessing, it lacks deeper algorithmic exploration and validation on how to fundamentally design improved architectures that are robust against interference and can genuinely capture spatio-temporal intrinsic forgery artifacts.

  2. The generators and datasets tested in this study are mainly concentrated in specific subsets; when facing rapidly evolving emerging technologies such as streaming generation or interactive real-time generation, the generalizability of its bias analysis still requires wider extended testing.

  3. The method fails to conduct comparative testing on current field SOTA (state-of-the-art) datasets such as GenVidBench, making it difficult to comprehensively evaluate the performance of this method under broader advanced benchmarks and weakening the persuasiveness of the experimental conclusions.

  4. The paper primarily focuses on the auditing and testing of existing methods; although it points out the advantages of frequency-domain methods, it fails to provide a comprehensive and integrated new benchmark to completely replace existing flawed benchmarks.

  5. In evaluating the impact of factors such as image/video spatial augmentation and resolution mismatch, some experimental analysis is relatively brief and fails to fully distinguish the complex compound effects of various concurrent biases intertwined with one another.

Questions for authors

see the above

Comments on the paper

1 comment

Dear reviewers,

The discussion phase is nearing its end; could you please respond to the authors?

Thank you & kind regards,

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.