- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces a dataset, GenTell-Gaze, that provides spatio-temporal anomaly tubes (trajectories of bounding boxes) and corresponding human gaze data for AI-generated videos. The proposed benchmark (GenTell-Bench) evaluates video classification, anomaly count and type prediction, temporal localization, and spatio-temporal localization. Experimental results show that open-weight and frontier VLMs underperform humans in real-vs.-AI classification and exhibit weak anomaly localization performance. When models use human gaze data, it improves the temporal localization performance of some VLMs but does not show consistent improvements overall.
Critical strengths
-
Practical and insightful combination of human gaze data and annotations of physical artifacts in AI-generated videos. This annotation potentially enables us to understand how humans identify AI-generated videos and why they classify them that way.
-
Interesting finding regarding the number of anomalies. Humans can more easily classify AI-generated videos with a larger number of artifacts, but VLMs do not show the same clear trend. This could provide useful insights for future model development.
Critical weaknesses
-
Limited analysis of human gaze data. How much do humans actually look at anomaly regions? Does looking at these regions help them correctly identify AI-generated videos?
-
Gaze patterns might depend on individual participant characteristics, but these are not presented in the paper. For example, age, gender, familiarity with AI or AI-generated images/videos, eyesight, etc. Although I understand that the authors did not collect personal or demographic data, some of these factors should have been addressed, at least in the participant recruitment criteria.
Questions for authors
-
How often did participants actually look at the annotated anomaly regions? Is there any relationship between gaze–anomaly alignment and the accuracy of human AI-generated video detection?
-
Could the authors clarify the participant recruitment and screening criteria, particularly regarding eyesight and familiarity with AI-generated content? How might individual differences (e.g., age, gender, or prior AI experience) affect the gaze patterns and the interpretation of the results?
2 comments
Dear Reviewer,
We have now uploaded a revised version of the paper. Additional gaze analysis is included in Section 5.1 and Appendix I, and a summary table of the participants' characteristics available to us, including a confirmation that these do not impact gaze data in a non-negligible manner, has been added to Appendix E.
Many thanks!
Thank you very much for your time and effort evaluating our work. Answers below.
We did run a more in-depth analysis of the human gaze before submitting the paper (it was actually one of the first things we did) but decided not to include it in the paper. Clearly, the reviewer indicated there is much interest, so these will be included. For your questions: On the first watch, people, on average, fixated on 33% of the annotated anomalies. Yes, this does help, as people whose gaze was fixed on the anomaly, meaning they saw it (which does not guarantee they realised it was an anomaly), had a higher accuracy at correctly detecting AI videos (66% when they didn’t fixate on an anomaly, 78% when they fixated in one watch, and 83% when they fixated on the same anomaly on both watches). The remaining 17% in the latter are people who saw the anomaly but must have decided it was not an anomaly, similar to the phenomenon we observed with Astra.
Regarding demographics, we did not collect race/nationally/etc, but we did make a record of gender, age and eyesight, as well as prior experience with generative AI. We have now checked all these characteristics against both accuracy and gaze on anomalies and found no detectable effect. Participants were recruited via flyers within the university and were mostly students, although there were few older participants. All participants had normal or corrected-to-normal vision. No participants were screened based on these criteria. We will add summary statistics of these characteristics and the effect analysis to the paper as well.