- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents RA-Bench, a benchmark designed to evaluate the reliability of AI-generated video (AIGV) detectors in real-world crisis events and other socially consequential scenarios. The benchmark contains 17,886 videos, including 1,830 real-video anchors and 16,056 generated videos produced by four open-source and five closed-source video generators.
Using RA-Bench, the authors conduct an extensive empirical evaluation of traditional forensic detectors, zero-shot multimodal large language models (MLLMs), and fine-tuned MLLM-based detectors. The study investigates cross-generator generalization, generation quality, conditioning strategies, sampling seeds, human authenticity judgments, and robustness under simulated social dissemination.
The main findings suggest that existing detectors exhibit limited generalization across generation sources, that generation characteristics influence detector performance, and that social dissemination operations further undermine detection reliability. The paper also identifies a noteworthy issue in the Skyra detector: its predictions appear sensitive to temporal-label representations, suggesting that evaluation protocol artifacts may introduce spurious cues unrelated to actual video authenticity.
Overall, the paper addresses a practically important problem and presents substantial empirical work. However, I have significant concerns regarding the novelty of its main findings, the distinctive scientific contribution of the proposed benchmark, and the depth of the analysis. In my view, these limitations prevent the paper from making a sufficiently compelling contribution to ICLR.
Critical strengths
S1. A practically relevant research problem.
The detection of AI-generated videos depicting wars, natural disasters, public emergencies, and other socially consequential events is an important problem. Investigating the reliability of existing detectors in such scenarios has clear practical relevance, particularly given the increasing realism of modern video generation systems.
S2. Broad empirical coverage and substantial experimental effort.
The paper evaluates multiple families of detectors, including traditional forensic models, zero-shot MLLMs, and fine-tuned MLLMs. It also considers several factors beyond conventional detection accuracy, such as generation conditions, human perception, and simulated social dissemination. The overall experimental coverage is relatively broad and reflects considerable effort.
S3. A potentially useful benchmark with real-event grounding.
RA-Bench provides a collection of real-event-conditioned generated videos across multiple social-risk categories and generation sources. Its paired construction strategy, human authenticity evaluation, and dissemination simulation may provide useful resources for subsequent research on AI-generated video detection.
S4. A noteworthy finding regarding spurious temporal-label priors.
One of the most interesting contributions is the analysis of Skyra's sensitivity to temporal-label representations. The authors show that replacing timestamps with frame indices, while keeping the visual frames unchanged, leads to a substantial reduction in detection performance.
This finding is particularly valuable because it suggests that a detector specifically trained for AIGV detection may rely on protocol-dependent shortcuts rather than genuine forensic evidence. Unlike several other observations in the paper, this result provides a relatively distinctive insight into detector reliability and the potential influence of spurious correlations.
I consider this analysis a meaningful strength of the paper.
Critical weaknesses
W1. Limited novelty of the main empirical findings.
My primary concern is that many of the paper's central conclusions largely reproduce phenomena that have already been extensively discussed in the AI-generated media forensics literature.
For example, the paper reports that:
Existing detectors exhibit poor generalization across unseen generation sources.
Higher-quality synthetic videos tend to be more difficult to detect, although the relationship varies across detector families.
Human-deceptive generated videos are also challenging for automated detectors.
Common video transformations associated with social dissemination can substantially degrade detection performance.
While these observations are practically relevant, the underlying problems of cross-generator generalization, detection robustness, and sensitivity to post-processing have been investigated extensively in prior research.
The paper mainly demonstrates that these previously recognized limitations persist on another benchmark involving socially consequential video content.
I acknowledge that empirical studies do not necessarily need to introduce new algorithms. However, their scientific contributions should be established through sufficiently novel observations, deeper explanations of existing phenomena, or previously unrecognized failure mechanisms.
In its current form, the paper provides considerable experimental evidence but relatively limited new scientific understanding beyond existing knowledge.
The temporal-label prior identified in Skyra is a notable exception. Unfortunately, most of the other conclusions do not demonstrate a comparable level of novelty.
W2. Insufficient justification of the distinctive scientific value of RA-Bench.
The introduction of RA-Bench constitutes a contribution, but I am not fully convinced that its scientific significance has been sufficiently established.
The authors emphasize that RA-Bench differs from existing benchmarks by focusing on real-world crisis events, employing real-video anchors, and incorporating human-deceptive and social dissemination evaluations.
These characteristics distinguish the benchmark in terms of data composition and evaluation coverage. However, the paper does not adequately demonstrate whether crisis-related scenarios introduce fundamentally different detection challenges compared with general-purpose AI-generated video datasets.
For example, it would be informative to compare detector behavior between crisis-related and ordinary video scenarios while controlling for generation sources, video quality, motion characteristics, and other potentially confounding factors.
Without such comparisons, it remains unclear whether the observed detection failures arise from distinctive properties of socially consequential content or simply reflect the well-known generalization limitations of existing detectors.
More broadly, a new benchmark should ideally enable the investigation of scientific questions that cannot be adequately addressed by existing resources, or reveal important previously unobserved phenomena.
Although RA-Bench extends scenario coverage and evaluation settings, the paper does not sufficiently establish this stronger form of contribution.
I therefore view its benchmark contribution as useful but primarily incremental.
W3. Broad experimental coverage but insufficient analytical depth.
Although the paper contains numerous experiments, many analyses remain primarily descriptive.
For instance, Section 4.2 investigates associations between generation quality, conditioning strategies, motion characteristics, and detector performance. The results indicate that traditional detectors and MLLM-based detectors respond differently to these factors.
However, the paper does not sufficiently explain the underlying reasons for these differences.
Do traditional forensic detectors rely more heavily on low-level spatial or temporal artifacts, while MLLM-based detectors respond more strongly to semantic or perceptual consistency? Are the observed differences attributable to model architectures, training objectives, data distributions, or evaluation protocols?
The current experiments do not provide sufficient evidence to distinguish among these possibilities.
Similarly, the finding that additional image conditioning affects different detector families differently is interesting, but the analysis does not establish the mechanisms responsible for these contrasting behaviors.
I do not expect an empirical study to provide a complete causal explanation for every observation. Nevertheless, when the main contribution of a paper is its analysis rather than a new algorithm, the depth and interpretability of that analysis become particularly important.
The paper would be substantially stronger if it moved beyond reporting performance differences and provided more systematic investigations into why these differences occur.
W4. Potential confounding factors in benchmark construction and evaluation.
The real-anchor-based generation strategy is a reasonable design choice, but it may also introduce systematic differences between real and generated videos that are unrelated to authenticity.
For example, although the authors standardize video encoding, generator-native resolution and frame rate are preserved. The real videos also originate from heterogeneous recording and dissemination environments.
Consequently, real and generated videos may differ in acquisition characteristics, temporal properties, resolution distributions, or other non-forensic attributes.
These differences could potentially influence detector predictions and complicate the interpretation of cross-generator performance.
The authors' own finding regarding Skyra's sensitivity to temporal-label representations illustrates the importance of carefully controlling such protocol-dependent cues.
This raises a broader question: to what extent might other detectors also exploit unintended correlations introduced by the benchmark construction or evaluation pipeline?
Additional controlled experiments that normalize relevant video properties, together with systematic analyses of potential dataset shortcuts, would strengthen the reliability of the conclusions.
I am not claiming that the reported results are invalid. Rather, the current evidence does not sufficiently rule out alternative explanations involving benchmark-specific confounding factors.
W5. The paper does not sufficiently capitalize on its most distinctive finding.
The analysis of Skyra's temporal-label sensitivity is arguably one of the paper's most scientifically interesting results.
It raises an important question: are specialized AIGV detectors learning genuine forensic evidence, or are they partially relying on shortcuts associated with training data and evaluation protocols?
This question is directly relevant to representation learning, shortcut learning, and out-of-distribution generalization.
However, the paper treats this finding as one component of a much broader empirical study rather than developing it into a systematic investigation.
For example, the authors could examine whether similar protocol-dependent behavior occurs in additional detectors, investigate the relationship between training data and learned shortcuts, or explore the conditions under which such dependencies emerge.
A deeper investigation could potentially transform this observation into a more generalizable insight about the reliability of learned forensic representations.
As currently presented, the finding is valuable but insufficiently developed relative to its potential significance.
Questions for authors
Q1. What are the most important scientific findings enabled by RA-Bench that could not have been obtained using existing AI-generated video benchmarks? Beyond additional scenario coverage, what fundamentally new understanding does RA-Bench provide?
Q2. Have the authors investigated whether crisis-related content presents distinct detection challenges compared with ordinary video content, after controlling for generation sources and relevant video characteristics?
Q3. Could the authors provide additional controlled evaluations to determine whether differences in resolution, frame rate, duration, or acquisition characteristics influence the reported detector performance?
Q4. The temporal-label sensitivity observed in Skyra is particularly interesting. Have the authors investigated whether other MLLM-based detectors exhibit similar dependencies on input formatting or evaluation protocols?
Q5. Can the authors provide deeper evidence explaining why traditional detectors and MLLM-based detectors respond differently to generation quality, motion, and conditioning strategies? In particular, can these differences be connected to the types of forensic evidence used by different detector families?
9 comments
Response to Reviewer Maag (2/8): Part 2, W2, Q1, Q2
Part 2. The distinctive value of RA-Bench (W2, Q1, Q2)
W2. Insufficient justification of the distinctive scientific value of RA-Bench.
Thank you for these comments. W2, Q1, and Q2 raise two questions: what RA-Bench enables that existing benchmarks do not, and whether crisis content is itself harder to detect. We answer them in turn.
Q1. What are the most important scientific findings enabled by RA-Bench that could not have been obtained using existing AI-generated video benchmarks? Beyond additional scenario coverage, what fundamentally new understanding does RA-Bench provide?
(1) RA-Bench enables findings that depend on its design, not only on its scenarios.
The table lists three findings, the design of RA-Bench that each relies on, and the key evidence. To our knowledge, existing benchmarks do not provide these designs together. Parts 1 and 5 give the details.
| Finding | Enabled by | Evidence |
|---|---|---|
| A news badge sways MLLM-based detectors | A dissemination protocol that includes a presentation edit and applies each operation separately | Gemini FakeR −28.1 to −32.4 points |
| Skyra depends on temporal labels | Generated durations that vary with the real clip, plus a fixed-duration control | BAcc 68.5% → 54.4% |
| The prompt format changes the class preference of zero-shot MLLMs | Three review settings (Binary, Diagnostic, Rating) applied to the same frames | Qwen3.5-0.8B FakeR 19.7% → 97.8%, Qwen3.5-4B 85.2% → 5.5% |
Beyond these findings, RA-Bench has a lasting value: it is continuously extensible. With the anchors, prompts, and protocol fixed, a newly released generator needs only one inference pass and is compared with earlier ones on the same anchors, as we showed with Wan3.0 during the rebuttal (Appendix F.1, and our response to Reviewer Tx3r). RA-Bench can therefore keep representing the state of video generation and keep guiding the development of detectors.
Q2. Have the authors investigated whether crisis-related content presents distinct detection challenges compared with ordinary video content, after controlling for generation sources and relevant video characteristics?
(2) Our focus on crisis content is motivated by consequence, and we added the control you suggest.
- Position. The paper focuses on crisis scenarios because of consequence rather than difficulty. It does not claim that crisis content is harder to detect.
- User study. In a small study with 20 volunteers, the RA-Bench video was judged more unsettling than an ordinary-scene generated video in 184 of 200 judgments (92.0%).
- Control. Following your suggestion and that of Reviewer c6nE, we compared RA-Bench with 500 ordinary-scene videos under the same generators and processing. Two findings follow.
- Crisis content is not noticeably harder to separate. The AUC of traditional detectors and the AUC and BAcc of Gemini are close between the two.
- What differs is the decision bias of Gemini. On RA-Bench it labels more videos as real, so its recall on real videos is higher and its recall on generated videos is lower, while its BAcc is nearly unchanged.
The settings and full results of the user study and the control are in our response to Reviewer c6nE, and the control has been added to the revision as Appendix F.7 (Table 41).
Response to Reviewer Maag (3/8): Part 3, W3 and Q5
Part 3. Why the detector families behave differently (W3, Q5)
W3. Broad experimental coverage but insufficient analytical depth.
Q5. Can the authors provide deeper evidence explaining why traditional detectors and MLLM-based detectors respond differently to generation quality, motion, and conditioning strategies? In particular, can these differences be connected to the types of forensic evidence used by different detector families?
Thank you for these specific questions. W3 and Q5 ask three things: what evidence each family relies on, what the differences can be attributed to, and why image conditioning affects the families differently. We answer them in turn. The analysis builds on the associations in Section 4.2 (Tables 3 and 22) and adds the design of each detector and controlled interventions.
(1) The two families rely on evidence at different levels, which explains their different responses.
The table summarizes how each family responds to the factors named in Q5. The numbers for imaging quality, consistency, and motion are changes in fake evidence from Table 22.
| Factor in Q5 | Traditional detectors (signal-level evidence) | MLLM detectors (semantic evidence) |
|---|---|---|
| Imaging quality | Affected. Higher imaging quality lowers the fake score (−9.4), so cleaner frames look more real. | Hardly affected (Gemini +0.6 to +1.8). |
| Subject and background consistency | Affected (−7.6 to −10.1). | Affected (Gemini −9.3 to −14.9). |
| Motion | Not affected (+0.1). | Affected. Dynamic clips yield more fake evidence (Gemini +5.5 to +8.4), likely because motion exposes more semantic errors. |
| Image conditioning | Affected. The score follows the real first frame, and the mean AUC moves from 33.4% (T2V) to 50.9% (first-frame I2V). | Affected. A real frame leaves fewer semantic errors, and the FakeR of the fine-tuned MLLMs falls from 70.5% to 42.7% and 28.3%. |
Two controlled interventions further indicate that the families respond to different levels of evidence.
- Signal-level changes strongly affect traditional detectors. Unifying the resolution moves UnivFD by +13.9 AUC points, and one transcode moves ForgeLens by −16.9. For Gemini, downsampling lowers BAcc by 2.3–4.7 points relative to T1, mainly by labeling more real videos as generated, while FakeR changes by at most 1.4 points.
- A presentation-level change shows the opposite pattern. The news badge leaves the traditional detectors nearly unchanged (mean AUC 48.5% vs. 49.1%) but lowers Gemini's FakeR by 28.1–32.4 points.
Response to Reviewer Maag (1/8): Part 1, W1
We sincerely thank you for the thorough and constructive review. You recognize the practical importance of the problem, the breadth of the experiments, and the value of the analysis of Skyra's temporal-label sensitivity, and you also raise very insightful questions. We have added experiments and analyses accordingly. Since your five weaknesses and five questions are closely related, we group our responses into five parts.
Part 1. Novelty of the main findings (W1)
Thank you for this careful examination. We agree that an empirical study should contribute novel observations, deeper explanations of existing phenomena, or previously unrecognized failure mechanisms. Rather than restating the four results, we clarify the novelty of the paper at two levels: the overall setting, and the controlled designs that turn known phenomena into specific findings.
(1) The overall setting is new.
(2) Phenomena discussed in prior work are examined systematically, which leads to specific findings.
We agree that some of these phenomena have been discussed before. What has been missing is a controlled examination that shows which factor is responsible. Three examples follow, and the effects of generation quality and conditioning are analyzed in Part 3.
Summary. We hope this shows that the paper offers more than evidence that "previously recognized limitations persist on another benchmark". The temporal-label finding is one outcome of this design, and the findings on the news badge and on human-deceptive videos are others. We have made this thread explicit in the revision: Appendix F.5 (Table 38) lists the factors, the controls, and the affected detectors.