acceptodds
Under review as a conference paper at ICLR 2027

Can We Defend Against AI-Generated Video Attacks on Real-World Crisis Events?

17 views

Abstract

Recent video generators can fabricate realistic depictions of wars, disasters, public emergencies, and other real-world crises, creating substantial misinformation risks. Existing benchmarks, however, provide limited evidence on detector and generator behavior in such settings, including how detectability varies with generation conditions, how people perceive generated videos, and whether detectors remain reliable during social dissemination. To address this gap, we introduce RA-Bench, a benchmark for AI-generated video detection that uses Real videos as Anchors. RA-Bench contains 17,886 videos, comprising 1,830 real-video anchors across 10 social-risk categories and 16,056 generated clips from four open-source and five closed-source generators. We assess detector generalization across seven traditional detectors, ten zero-shot multimodal models under three review settings, and two MLLMs fine-tuned for AI-generated video detection. None of the three detector families generalizes consistently across RA-Bench instances. We then examine generation quality, conditioning information, and sampling seeds. Generation properties affect detector families differently, while source-level detection patterns for the four open-source generators remain stable across seeds. Finally, we study human authenticity judgments and detector reliability during social dissemination. We find that videos that mislead people are also difficult for current detectors, and that social dissemination makes detection harder. Together, these findings show that current methods struggle to detect realistic AI-generated videos, highlighting the need for detectors robust to evolving video generators.

Primary area:
datasets and benchmarks
Keywords:
AI-generated video detection, benchmark, multimodal large language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_MaagOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents RA-Bench, a benchmark designed to evaluate the reliability of AI-generated video (AIGV) detectors in real-world crisis events and other socially consequential scenarios. The benchmark contains 17,886 videos, including 1,830 real-video anchors and 16,056 generated videos produced by four open-source and five closed-source video generators.

Using RA-Bench, the authors conduct an extensive empirical evaluation of traditional forensic detectors, zero-shot multimodal large language models (MLLMs), and fine-tuned MLLM-based detectors. The study investigates cross-generator generalization, generation quality, conditioning strategies, sampling seeds, human authenticity judgments, and robustness under simulated social dissemination.

The main findings suggest that existing detectors exhibit limited generalization across generation sources, that generation characteristics influence detector performance, and that social dissemination operations further undermine detection reliability. The paper also identifies a noteworthy issue in the Skyra detector: its predictions appear sensitive to temporal-label representations, suggesting that evaluation protocol artifacts may introduce spurious cues unrelated to actual video authenticity.

Overall, the paper addresses a practically important problem and presents substantial empirical work. However, I have significant concerns regarding the novelty of its main findings, the distinctive scientific contribution of the proposed benchmark, and the depth of the analysis. In my view, these limitations prevent the paper from making a sufficiently compelling contribution to ICLR.

Critical strengths

S1. A practically relevant research problem.

The detection of AI-generated videos depicting wars, natural disasters, public emergencies, and other socially consequential events is an important problem. Investigating the reliability of existing detectors in such scenarios has clear practical relevance, particularly given the increasing realism of modern video generation systems.

S2. Broad empirical coverage and substantial experimental effort.

The paper evaluates multiple families of detectors, including traditional forensic models, zero-shot MLLMs, and fine-tuned MLLMs. It also considers several factors beyond conventional detection accuracy, such as generation conditions, human perception, and simulated social dissemination. The overall experimental coverage is relatively broad and reflects considerable effort.

S3. A potentially useful benchmark with real-event grounding.

RA-Bench provides a collection of real-event-conditioned generated videos across multiple social-risk categories and generation sources. Its paired construction strategy, human authenticity evaluation, and dissemination simulation may provide useful resources for subsequent research on AI-generated video detection.

S4. A noteworthy finding regarding spurious temporal-label priors.

One of the most interesting contributions is the analysis of Skyra's sensitivity to temporal-label representations. The authors show that replacing timestamps with frame indices, while keeping the visual frames unchanged, leads to a substantial reduction in detection performance.

This finding is particularly valuable because it suggests that a detector specifically trained for AIGV detection may rely on protocol-dependent shortcuts rather than genuine forensic evidence. Unlike several other observations in the paper, this result provides a relatively distinctive insight into detector reliability and the potential influence of spurious correlations.

I consider this analysis a meaningful strength of the paper.

Critical weaknesses

W1. Limited novelty of the main empirical findings.

My primary concern is that many of the paper's central conclusions largely reproduce phenomena that have already been extensively discussed in the AI-generated media forensics literature.

For example, the paper reports that:

Existing detectors exhibit poor generalization across unseen generation sources.

Higher-quality synthetic videos tend to be more difficult to detect, although the relationship varies across detector families.

Human-deceptive generated videos are also challenging for automated detectors.

Common video transformations associated with social dissemination can substantially degrade detection performance.

While these observations are practically relevant, the underlying problems of cross-generator generalization, detection robustness, and sensitivity to post-processing have been investigated extensively in prior research.

The paper mainly demonstrates that these previously recognized limitations persist on another benchmark involving socially consequential video content.

I acknowledge that empirical studies do not necessarily need to introduce new algorithms. However, their scientific contributions should be established through sufficiently novel observations, deeper explanations of existing phenomena, or previously unrecognized failure mechanisms.

In its current form, the paper provides considerable experimental evidence but relatively limited new scientific understanding beyond existing knowledge.

The temporal-label prior identified in Skyra is a notable exception. Unfortunately, most of the other conclusions do not demonstrate a comparable level of novelty.

W2. Insufficient justification of the distinctive scientific value of RA-Bench.

The introduction of RA-Bench constitutes a contribution, but I am not fully convinced that its scientific significance has been sufficiently established.

The authors emphasize that RA-Bench differs from existing benchmarks by focusing on real-world crisis events, employing real-video anchors, and incorporating human-deceptive and social dissemination evaluations.

These characteristics distinguish the benchmark in terms of data composition and evaluation coverage. However, the paper does not adequately demonstrate whether crisis-related scenarios introduce fundamentally different detection challenges compared with general-purpose AI-generated video datasets.

For example, it would be informative to compare detector behavior between crisis-related and ordinary video scenarios while controlling for generation sources, video quality, motion characteristics, and other potentially confounding factors.

Without such comparisons, it remains unclear whether the observed detection failures arise from distinctive properties of socially consequential content or simply reflect the well-known generalization limitations of existing detectors.

More broadly, a new benchmark should ideally enable the investigation of scientific questions that cannot be adequately addressed by existing resources, or reveal important previously unobserved phenomena.

Although RA-Bench extends scenario coverage and evaluation settings, the paper does not sufficiently establish this stronger form of contribution.

I therefore view its benchmark contribution as useful but primarily incremental.

W3. Broad experimental coverage but insufficient analytical depth.

Although the paper contains numerous experiments, many analyses remain primarily descriptive.

For instance, Section 4.2 investigates associations between generation quality, conditioning strategies, motion characteristics, and detector performance. The results indicate that traditional detectors and MLLM-based detectors respond differently to these factors.

However, the paper does not sufficiently explain the underlying reasons for these differences.

Do traditional forensic detectors rely more heavily on low-level spatial or temporal artifacts, while MLLM-based detectors respond more strongly to semantic or perceptual consistency? Are the observed differences attributable to model architectures, training objectives, data distributions, or evaluation protocols?

The current experiments do not provide sufficient evidence to distinguish among these possibilities.

Similarly, the finding that additional image conditioning affects different detector families differently is interesting, but the analysis does not establish the mechanisms responsible for these contrasting behaviors.

I do not expect an empirical study to provide a complete causal explanation for every observation. Nevertheless, when the main contribution of a paper is its analysis rather than a new algorithm, the depth and interpretability of that analysis become particularly important.

The paper would be substantially stronger if it moved beyond reporting performance differences and provided more systematic investigations into why these differences occur.

W4. Potential confounding factors in benchmark construction and evaluation.

The real-anchor-based generation strategy is a reasonable design choice, but it may also introduce systematic differences between real and generated videos that are unrelated to authenticity.

For example, although the authors standardize video encoding, generator-native resolution and frame rate are preserved. The real videos also originate from heterogeneous recording and dissemination environments.

Consequently, real and generated videos may differ in acquisition characteristics, temporal properties, resolution distributions, or other non-forensic attributes.

These differences could potentially influence detector predictions and complicate the interpretation of cross-generator performance.

The authors' own finding regarding Skyra's sensitivity to temporal-label representations illustrates the importance of carefully controlling such protocol-dependent cues.

This raises a broader question: to what extent might other detectors also exploit unintended correlations introduced by the benchmark construction or evaluation pipeline?

Additional controlled experiments that normalize relevant video properties, together with systematic analyses of potential dataset shortcuts, would strengthen the reliability of the conclusions.

I am not claiming that the reported results are invalid. Rather, the current evidence does not sufficiently rule out alternative explanations involving benchmark-specific confounding factors.

W5. The paper does not sufficiently capitalize on its most distinctive finding.

The analysis of Skyra's temporal-label sensitivity is arguably one of the paper's most scientifically interesting results.

It raises an important question: are specialized AIGV detectors learning genuine forensic evidence, or are they partially relying on shortcuts associated with training data and evaluation protocols?

This question is directly relevant to representation learning, shortcut learning, and out-of-distribution generalization.

However, the paper treats this finding as one component of a much broader empirical study rather than developing it into a systematic investigation.

For example, the authors could examine whether similar protocol-dependent behavior occurs in additional detectors, investigate the relationship between training data and learned shortcuts, or explore the conditions under which such dependencies emerge.

A deeper investigation could potentially transform this observation into a more generalizable insight about the reliability of learned forensic representations.

As currently presented, the finding is valuable but insufficiently developed relative to its potential significance.

Questions for authors

Q1. What are the most important scientific findings enabled by RA-Bench that could not have been obtained using existing AI-generated video benchmarks? Beyond additional scenario coverage, what fundamentally new understanding does RA-Bench provide?

Q2. Have the authors investigated whether crisis-related content presents distinct detection challenges compared with ordinary video content, after controlling for generation sources and relevant video characteristics?

Q3. Could the authors provide additional controlled evaluations to determine whether differences in resolution, frame rate, duration, or acquisition characteristics influence the reported detector performance?

Q4. The temporal-label sensitivity observed in Skyra is particularly interesting. Have the authors investigated whether other MLLM-based detectors exhibit similar dependencies on input formatting or evaluation protocols?

Q5. Can the authors provide deeper evidence explaining why traditional detectors and MLLM-based detectors respond differently to generation quality, motion, and conditioning strategies? In particular, can these differences be connected to the types of forensic evidence used by different detector families?

9 comments
AuthorsOpenReview ↗

Response to Reviewer Maag (1/8): Part 1, W1

We sincerely thank you for the thorough and constructive review. You recognize the practical importance of the problem, the breadth of the experiments, and the value of the analysis of Skyra's temporal-label sensitivity, and you also raise very insightful questions. We have added experiments and analyses accordingly. Since your five weaknesses and five questions are closely related, we group our responses into five parts.

Part 1. Novelty of the main findings (W1)

W1. Limited novelty of the main empirical findings. "[...] their scientific contributions should be established through sufficiently novel observations, deeper explanations of existing phenomena, or previously unrecognized failure mechanisms. [...] The temporal-label prior identified in Skyra is a notable exception."

Thank you for this careful examination. We agree that an empirical study should contribute novel observations, deeper explanations of existing phenomena, or previously unrecognized failure mechanisms. Rather than restating the four results, we clarify the novelty of the paper at two levels: the overall setting, and the controlled designs that turn known phenomena into specific findings.

(1) The overall setting is new.

  • Focus. Existing benchmarks are built mainly from general-purpose video content, and none of those in Table 1 of the paper is fully grounded in real events. RA-Bench targets real-world crises and other socially consequential events, where a forged video can cause the most harm.
  • Scenario. It studies a concrete misinformation scenario in which a real frame from such an event is used to generate what happens next. Every real anchor is paired with clips from nine recent generators, five of them closed-source.
  • Scope of evaluation. It couples detection with human judgments and a dissemination simulation. In that table, no existing benchmark provides a human-deceptive challenge set, and dissemination is covered at most partially.

(2) Phenomena discussed in prior work are examined systematically, which leads to specific findings.

We agree that some of these phenomena have been discussed before. What has been missing is a controlled examination that shows which factor is responsible. Three examples follow, and the effects of generation quality and conditioning are analyzed in Part 3.

  • Dissemination. Robustness to post-processing has been studied before, but the individual and combined effects of these factors have rarely been examined systematically. LastMile evaluates transcoding, downsampling, frame-rate reduction, and a presentation edit separately and together in one protocol. Including the news badge revealed a tendency of MLLM-based detectors: a mark that leaves the scene untouched lowers Gemini's FakeR by 28.1–32.4 points. We believe that a model judging by semantic content is strongly swayed by a cue that usually accompanies authentic footage and then loses its grip on the details that decide authenticity.
  • Video duration. Generators have default durations, so training and evaluation data tend to share them (99.9% of the generated videos in ViF-Bench are exactly 5.00 s), and such an evaluation set rewards a duration shortcut instead of exposing it. RA-Bench varies the generated duration with the real clip and adds a fixed-duration control, which is how Skyra's dependence on temporal labels surfaced. We then isolated it with two prompts that differ only in the timestamp format. A benchmark should therefore vary the attributes that generators leave constant.
  • Human deception. Beyond average human accuracy, we collect per-video judgments and ask whether detectors fail on the same videos as people. On the 633 generated videos judged real by all five raters, Gemini's FakeR is 15.6–17.3 points below the source-matched reference, whereas traditional detectors change only slightly. Detectors that judge by semantic content thus behave more like human viewers.

Summary. We hope this shows that the paper offers more than evidence that "previously recognized limitations persist on another benchmark". The temporal-label finding is one outcome of this design, and the findings on the news badge and on human-deceptive videos are others. We have made this thread explicit in the revision: Appendix F.5 (Table 38) lists the factors, the controls, and the affected detectors.

AuthorsOpenReview ↗

Response to Reviewer Maag (2/8): Part 2, W2, Q1, Q2

Part 2. The distinctive value of RA-Bench (W2, Q1, Q2)

W2. Insufficient justification of the distinctive scientific value of RA-Bench.

Thank you for these comments. W2, Q1, and Q2 raise two questions: what RA-Bench enables that existing benchmarks do not, and whether crisis content is itself harder to detect. We answer them in turn.

Q1. What are the most important scientific findings enabled by RA-Bench that could not have been obtained using existing AI-generated video benchmarks? Beyond additional scenario coverage, what fundamentally new understanding does RA-Bench provide?

(1) RA-Bench enables findings that depend on its design, not only on its scenarios.

The table lists three findings, the design of RA-Bench that each relies on, and the key evidence. To our knowledge, existing benchmarks do not provide these designs together. Parts 1 and 5 give the details.

FindingEnabled byEvidence
A news badge sways MLLM-based detectorsA dissemination protocol that includes a presentation edit and applies each operation separatelyGemini FakeR −28.1 to −32.4 points
Skyra depends on temporal labelsGenerated durations that vary with the real clip, plus a fixed-duration controlBAcc 68.5% → 54.4%
The prompt format changes the class preference of zero-shot MLLMsThree review settings (Binary, Diagnostic, Rating) applied to the same framesQwen3.5-0.8B FakeR 19.7% → 97.8%, Qwen3.5-4B 85.2% → 5.5%

Beyond these findings, RA-Bench has a lasting value: it is continuously extensible. With the anchors, prompts, and protocol fixed, a newly released generator needs only one inference pass and is compared with earlier ones on the same anchors, as we showed with Wan3.0 during the rebuttal (Appendix F.1, and our response to Reviewer Tx3r). RA-Bench can therefore keep representing the state of video generation and keep guiding the development of detectors.

Q2. Have the authors investigated whether crisis-related content presents distinct detection challenges compared with ordinary video content, after controlling for generation sources and relevant video characteristics?

(2) Our focus on crisis content is motivated by consequence, and we added the control you suggest.

  • Position. The paper focuses on crisis scenarios because of consequence rather than difficulty. It does not claim that crisis content is harder to detect.
  • User study. In a small study with 20 volunteers, the RA-Bench video was judged more unsettling than an ordinary-scene generated video in 184 of 200 judgments (92.0%).
  • Control. Following your suggestion and that of Reviewer c6nE, we compared RA-Bench with 500 ordinary-scene videos under the same generators and processing. Two findings follow.
    • Crisis content is not noticeably harder to separate. The AUC of traditional detectors and the AUC and BAcc of Gemini are close between the two.
    • What differs is the decision bias of Gemini. On RA-Bench it labels more videos as real, so its recall on real videos is higher and its recall on generated videos is lower, while its BAcc is nearly unchanged.

The settings and full results of the user study and the control are in our response to Reviewer c6nE, and the control has been added to the revision as Appendix F.7 (Table 41).

AuthorsOpenReview ↗

Response to Reviewer Maag (3/8): Part 3, W3 and Q5

Part 3. Why the detector families behave differently (W3, Q5)

W3. Broad experimental coverage but insufficient analytical depth.

Q5. Can the authors provide deeper evidence explaining why traditional detectors and MLLM-based detectors respond differently to generation quality, motion, and conditioning strategies? In particular, can these differences be connected to the types of forensic evidence used by different detector families?

Thank you for these specific questions. W3 and Q5 ask three things: what evidence each family relies on, what the differences can be attributed to, and why image conditioning affects the families differently. We answer them in turn. The analysis builds on the associations in Section 4.2 (Tables 3 and 22) and adds the design of each detector and controlled interventions.

(1) The two families rely on evidence at different levels, which explains their different responses.

The table summarizes how each family responds to the factors named in Q5. The numbers for imaging quality, consistency, and motion are changes in fake evidence from Table 22.

Factor in Q5Traditional detectors (signal-level evidence)MLLM detectors (semantic evidence)
Imaging qualityAffected. Higher imaging quality lowers the fake score (−9.4), so cleaner frames look more real.Hardly affected (Gemini +0.6 to +1.8).
Subject and background consistencyAffected (−7.6 to −10.1).Affected (Gemini −9.3 to −14.9).
MotionNot affected (+0.1).Affected. Dynamic clips yield more fake evidence (Gemini +5.5 to +8.4), likely because motion exposes more semantic errors.
Image conditioningAffected. The score follows the real first frame, and the mean AUC moves from 33.4% (T2V) to 50.9% (first-frame I2V).Affected. A real frame leaves fewer semantic errors, and the FakeR of the fine-tuned MLLMs falls from 70.5% to 42.7% and 28.3%.

Two controlled interventions further indicate that the families respond to different levels of evidence.

  • Signal-level changes strongly affect traditional detectors. Unifying the resolution moves UnivFD by +13.9 AUC points, and one transcode moves ForgeLens by −16.9. For Gemini, downsampling lowers BAcc by 2.3–4.7 points relative to T1, mainly by labeling more real videos as generated, while FakeR changes by at most 1.4 points.
  • A presentation-level change shows the opposite pattern. The news badge leaves the traditional detectors nearly unchanged (mean AUC 48.5% vs. 49.1%) but lowers Gemini's FakeR by 28.1–32.4 points.
Reviewer_c6nEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the misinformation risk of AI-generated videos depicting real-world crises such as wars, disasters, and public emergencies, where existing benchmarks provide limited evidence on detector and generator behavior. It introduces RA-Bench, a benchmark that uses real videos as anchors, containing 17,886 videos — 1,830 real-video anchors across 10 social-risk categories and 16,056 generated clips from four open-source and five closed-source generators. It assesses detector generalization across seven traditional detectors, ten zero-shot MLLM detectors under three review settings, and two fine-tuned MLLMs, finding that none of the three detector families generalizes consistently. It further examines generation quality, conditioning information, and sampling seeds, showing that generation properties affect detector families differently while source-level patterns remain stable across seeds. Finally, it studies human authenticity judgments and detector reliability during social dissemination, finding that videos which mislead people are also difficult for current detectors, and that dissemination makes detection harder.

Critical strengths
  1. This paper identifies the AIGC detection to survive social dissemination, a large-scale benchmark grounded in real events (17,886 videos; 1,830 real anchors; 10 social-risk categories; 9 generators) is designed.

  2. The evaluation spans three detector families in one protocol — seven traditional detectors, ten zero-shot MLLM detectors under three review settings, and two fine-tuned MLLMs — supporting the central finding that no family generalizes consistently across sources.

  3. Beyond detector ranking, this paper analyzes generation properties (quality, conditioning, seeds) and couples machine detection with human judgment and a dissemination simulation, yielding decision-relevant findings: videos that mislead people are also hard for detectors, and social dissemination makes detection harder.

Critical weaknesses
  1. This paper lacks a comparison with ordinary scenes under the same generators and processing conditions, making it difficult to establish whether crisis scenarios introduce unique detection challenges.
  2. The generalizability of the dissemination results is limited. Four of the five configurations are Skyra variants, and the best-performing zero-shot detector is not evaluated. The predefined transformations are also not validated against actual platform processing pipelines, so broader conclusions about real-world social dissemination require further evidence.
Questions for authors

Please see the weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer c6nE (1/3): W1

We sincerely thank you for the positive assessment of our work. It is very encouraging that you recognize the evaluation of three detector families in one protocol and the decision-relevant findings obtained by coupling machine detection with human judgment and a dissemination simulation. Both of your comments are highly constructive, and we have conducted additional experiments accordingly.

W1: This paper lacks a comparison with ordinary scenes under the same generators and processing conditions, making it difficult to establish whether crisis scenarios introduce unique detection challenges.

Thank you for proposing this control. We first clarify our position and then report the control.

Our position. Our focus on crisis scenarios is motivated by consequence rather than by difficulty: a miss or a false alarm on crisis footage can cause serious societal harm. Whether crisis content is itself harder to detect is a separate and valuable question.

A small user study. To check this position, we sampled 100 generated crisis-scene videos from RA-Bench and 100 generated ordinary-scene videos from GenVidBench and formed 100 pairs. 20 newly recruited volunteers each judged 10 pairs, choosing which video made them feel more uneasy or reporting no difference, which gives 200 judgments in total.

Which video is more unsettling?RA-BenchGenVidBenchNo difference
Judgments (20 participants)184 (92.0%)6 (3.0%)10 (5.0%)

The RA-Bench video was chosen as more unsettling in 184 of the 200 judgments (92.0%). Although small, this study is consistent with our motivation that forged footage of crisis events is more disturbing to viewers than forged ordinary footage.

Following your suggestion, we have also run this control. Its setting is as follows.

  • Ordinary-scene videos. RA-Bench retains only real-world crises and other socially consequential events. During its construction, clips of ordinary scenes that did not fit this scope were excluded in manual review. For this control, we collected 500 ordinary-scene clips from this excluded footage, so they come from the same source collection as the RA-Bench anchors.

  • Same generators and processing. These videos go through exactly the same pipeline as RA-Bench: the same first-frame and prompt construction, the same generators, and the same encoding.

  • Same evaluation. We evaluate the seven traditional detectors and Gemini-3.1-Pro-Preview and collect human authenticity judgments, all with the protocol of the paper, and compare the results with those reported for RA-Bench.

    MetricRA-BenchOrdinary scenesDifference (points) [95% CI]
    Seven traditional detectors, mean AUC51.30%50.91%+0.39 [−0.74, +1.54]
    Seven traditional detectors, mean TPR at 5% FPR5.41%4.88%+0.53 [+0.09, +1.00]
    Gemini Rating, AUC63.61%63.76%−0.15 [−2.81, +2.34]
    Gemini Binary, balanced accuracy (BAcc)63.40%62.24%+1.16 [−0.40, +2.64]
    Gemini Binary, recall on real videos (RealR)72.36%62.82%+9.54 [+5.74, +13.58]
    Gemini Binary, recall on generated videos (FakeR)54.44%61.65%−7.21 [−10.15, −4.44]
    Human raters, BAcc65.84%66.35%−0.51 [−1.58, +0.48]

    Analysis. Two observations follow from the table.

    • Crisis content is not noticeably harder to separate. Traditional-detector AUC, Gemini Rating AUC, Gemini BAcc, and human BAcc differ by about 1 point or less (+0.39, −0.15, +1.16, −0.51), and all four 95% CIs include zero.
    • What differs is the decision bias of Gemini. On RA-Bench as a whole, Gemini Binary has a 9.5-point higher RealR and a 7.2-point lower FakeR than on the ordinary-scene clips (neither CI includes zero), while its BAcc is nearly unchanged. Human BAcc is essentially the same (65.84% vs. 66.35%).

    Conclusion. Crisis-centered content is not harder to separate, but it is associated with a shift of Gemini toward missing generated videos, which is the costly error in this setting. An explanation consistent with the current evidence is that the model treats the news-documentary context of such footage as a cue for authenticity, in the same direction as the news badge (see W2-1). This comparison has been added to the revision as Appendix F.7 (Table 41).

AuthorsOpenReview ↗

Response to Reviewer c6nE (2/3): W2-1

W2-1: The generalizability of the dissemination results is limited. Four of the five configurations are Skyra variants, and the best-performing zero-shot detector is not evaluated.

Thank you for raising this point. We first clarify what the submission already covers and then report the additional evaluation of Gemini.

  • Skyra configurations. The four Skyra configurations are a controlled test of the timestamp dependence. They compare two sets of weights under two temporal-label formats (Section 4.1).

  • Other detectors. Figure 5 of the submission also reports the seven traditional detectors and BusterX++ under the same LastMile conditions.

  • Added evaluation. Following your suggestion, we added Gemini-3.1-Pro-Preview, the best zero-shot detector in the main experiments. We use the three prompts and scoring rules of the main experiments under all six LastMile conditions (150 real and 1,350 generated videos per condition).

    Each cell reports BAcc / FakeR / RealR (%), i.e., balanced accuracy, recall on generated videos, and recall on real videos.

    ConditionBinaryDiagnosticRating
    Original64.3 / 55.9 / 72.763.3 / 50.7 / 76.063.7 / 53.4 / 74.0
    T1 (transcoding)62.6 / 57.8 / 67.362.1 / 50.2 / 74.062.0 / 52.7 / 71.3
    T1+T2 (downsampling)57.9 / 56.4 / 59.357.9 / 49.1 / 66.759.7 / 53.4 / 66.0
    T1+T3 (8 fps)63.4 / 58.1 / 68.762.3 / 52.0 / 72.758.6 / 51.9 / 65.3
    T1+T4 (news badge)58.4 / 26.8 / 90.058.0 / 22.1 / 94.056.1 / 20.3 / 92.0
    Full55.6 / 25.3 / 86.056.1 / 20.3 / 92.056.5 / 19.7 / 93.3

    The results show three things.

    • Robustness. Gemini is not robust to dissemination. Relative to transcoding alone (T1), its BAcc falls by 5.5–7.0 points under the full simulation, to 55.6–56.5%.
    • News badge. The news badge shifts its decisions the most. The badge occupies only a corner of the frame and leaves the content unchanged. Yet, relative to transcoding alone, FakeR drops by 28.1–32.4 points and RealR rises by 20.0–22.7 points under all three prompts. The fine-tuned MLLMs in the submission move in the same direction (FakeR from 29.9% to 14.0%).
    • Implication. Once the model sees a local mark that it associates with authentic footage, it relies less on the visual content itself. A presentation cue with no forensic information can steer the verdict, which is a caution for the design and evaluation of detectors.
AuthorsOpenReview ↗

Response to Reviewer c6nE (3/3): W2-2 and Summary

W2-2: The predefined transformations are also not validated against actual platform processing pipelines, so broader conclusions about real-world social dissemination require further evidence.

Thank you for this suggestion. The RA-Bench-LastMile settings correspond to real-world practice as follows.

  • News badge. A persistent corner logo is standard on news video from outlets such as the BBC and the Associated Press. It is also present in our own data: RA-Bench sources include AP and Reuters footage, and some real anchors already carry an upper-right badge (see the LastMile details in the appendix). Fabricated videos imitate this branding as well, as fact-checkers have documented for a fabricated BBC-branded broadcast [4].
  • Scaling and re-encoding. Reported resolution reductions of shared copies are about 0.5 on WhatsApp [2] and 0.64 and 0.8 on YouTube and Facebook [1]. The 0.5× downsampling in LastMile is at least as strong as the strongest of these, so LastMile is a stress setting rather than a model of a single platform. Platform compression parameters have likewise been estimated from actually uploaded and downloaded files [3].

A quantitative extrapolation to specific platforms requires paired files of the same videos before and after upload, which is the next step for LastMile.

Summary. Your two comments lead to two more specific conclusions for the paper. The crisis context does not change separability but shifts Gemini toward misses, and for Gemini, the main cause of missed generated videos under dissemination is not signal degradation but an authenticity-irrelevant badge. The RA-Bench-LastMile results of Gemini have been added to the revision (Appendix F.4, Table 37).

Thank you again for your recognition and constructive comments. We hope the responses above resolve your concerns.

References

[1] Marcon, Pasquini, Boato. Detection of Manipulated Face Videos over Social Networks: A Large-Scale Study. Journal of Imaging 7(10):193, 2021.

[2] Meij, Geradts. Source camera identification using Photo Response Non-Uniformity on WhatsApp. Digital Investigation 24:142–154, 2018.

[3] Montibeller, Shullani, Baracchi, Piva, Boato. Bridging the Gap: A Framework for Real-World Video Deepfake Detection via Social Network Compression Emulation. DFF '25, arXiv:2508.08765.

[4] Check Your Fact. Fact Check: Did BBC News Broadcast This Video Warning Of A Nuclear Attack? 2022.

Dear Reviewer c6nE,

Thank you for suggesting the ordinary-scene control and the dissemination evaluation beyond Skyra. We wanted to follow up on what those experiments showed.

  1. Crisis versus ordinary scenes (W1). We processed 500 ordinary-scene anchors from the same source collection using the same generators and encoding. Mean traditional-detector AUC and human balanced accuracy each differed by less than one point between settings. On crisis footage, however, Gemini Binary’s recall was 9.54 points higher for real videos and 7.21 points lower for generated videos (Table 41).

  2. Dissemination beyond Skyra (W2). We evaluated Gemini-3.1-Pro-Preview, our strongest zero-shot detector, under all six LastMile conditions. Adding a corner news badge to the same transcoded clips lowered its recall on generated videos by 28.1–32.4 points across the three prompts (Table 37).

  3. Connection to actual platforms (W2). Appendix F.4 relates our 0.5× downsampling to published measurements of shared video copies. We describe LastMile as a controlled stress test. Estimating the effect on a specific platform would require paired copies of the same videos before and after upload.

As the discussion draws to a close, we would welcome your feedback on whether these additions address your two concerns. Please let us know if anything needs further clarification.

Best regards,
Authors of Submission 15594

Reviewer_Tx3rOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on the systematic evaluation of defense capabilities in the field of AI-generated video detection. To address the shortcomings of existing benchmarks in source control, prompt alignment, and adversarial real-world scenarios, the authors construct a novel and challenging benchmark, systematically examining the robustness and generalization boundaries of various current mainstream detectors against advanced video generation models, thereby providing important experimental support and performance insights for future trustworthy multimedia forensics research.

Critical strengths
  1. The paper keenly points out the limitations of current AI-generated video detection benchmarks in variable control and real-world scenario alignment, and constructs a rigorously designed, comprehensive new evaluation benchmark with high academic foresight and practical value.

  2. Through large-scale experimental data and detailed analyses, the paper profoundly reveals the capability boundaries of current mainstream detectors when dealing with complex dynamics and generative artifacts, pointing out key directions for subsequently enhancing detection defense robustness.

Critical weaknesses

1.The overall scale of the dataset is relatively small, and the covered generators are also limited, which may lead to biased statistical results and make it difficult to comprehensively represent the current landscape of rapidly evolving video generation technologies.

2.The methods are not adequately compared with results on SOTA (state-of-the-art) datasets in the current field, such as GenVidBench and GenVido, thereby weakening the persuasiveness of the experimental conclusions.

3.There is a lack of performance comparisons with classic detection methods such as Demamba on this dataset, making it insufficient to fully demonstrate the advantages of the proposed dataset.

4.The evaluation and detection methods proposed in the paper focus primarily on specific fully-synthesized video scenarios, lacking sufficient coverage for more challenging real-world complex forgery forms such as localized manipulation, video splicing, or hybrid editing.

Questions for authors

see the weekness

5 comments
AuthorsOpenReview ↗

Response to Reviewer Tx3r (1/3): W1

We sincerely thank you for reviewing our work and for recognizing the design and practical value of the benchmark. Following your four comments, we have added a measured comparison with existing benchmarks and an extension to a new generator. Our point-by-point responses are below.

W1-1: The overall scale of the dataset is relatively small, and the covered generators are also limited

Thank you for raising this point. We first compare RA-Bench with the publicly released evaluation splits of existing benchmarks, and then answer in three points:

Evaluation split# VideosNewer generators
GenVideo-Val18,58456 Sora demo clips (Feb. 2024)
GenVidBench test set68,180Sora and Kling (2024), about 320 clips in total
RA-Bench17,886Kling 3.0, Seedance 2.0, and HappyHorse 1.0 (2026), 5,101 clips in total
  • Scale. RA-Bench is an evaluation-only benchmark, and its scale is of the same order as existing evaluation splits. The fair reference is therefore the evaluation split of each benchmark rather than a million-scale training corpus.
  • Data quality. The construction pipeline is rigorous and labor-intensive, which ensures high video quality.
    • From 675 public source videos we extracted 5,774 candidate clips. Two reviewers assessed each clip independently, and disagreements and uncertain cases were adjudicated in a second round.
    • After homogeneity pruning and a licensing review, 1,830 anchors were retained.
    • The first frame of each anchor and a single anchor-derived prompt are fed to all nine generators, so each generated video shares its scene and content with a real anchor (video–image subject and background consistency of 92.3–97.4, Table 21).
  • Generator coverage. The covered generators are newer. The two existing splits are dominated by models from 2024 or earlier, whereas RA-Bench covers generators released in 2026, and 54% of its generated videos (8,736 clips) come from closed-source generators.

W1-2: which may lead to biased statistical results and make it difficult to comprehensively represent the current landscape of rapidly evolving video generation technologies.

This concern has two parts, statistical bias and the coverage of new generators. We address them in turn.

(1) We examined possible statistical bias in two ways.

  • Sample size. The conclusions rest on 43,506 generated videos once the seed, duration, and conditioning controls are included. In addition, 17,850 videos were each judged independently by at least three human raters.
  • Seed stability. Across three random seeds, the seven-detector mean AUC of each of the four open-source generators varies by at most 1.05 points (Table 4).

(2) RA-Bench is designed to keep pace with rapidly evolving generators.

  • Continuous extension. No fixed generator list can stay current, so we designed RA-Bench to be continuously extended, which we regard as one of its main contributions.
    • The screened anchors, first frames, prompts, and evaluation protocol are all fixed.
    • Adding a generator requires only one inference pass over the existing first frames and prompts.
    • The new generator can be compared directly with earlier ones on the same anchors.
  • Demonstration with Wan3.0. We added Wan3.0 during the rebuttal. Due to time constraints, we currently use 500 of the 1,830 anchors (all 10 categories and 41 of the 44 subcategories), with the same first frames and prompts as Wan2.2.
Metric (%)Wan2.2Wan3.0
Seven traditional detectors, mean AUC48.145.9
ForgeLens, AUC56.846.2
DeCoF, AUC66.759.7
Skyra (four configurations), balanced accuracy60.1–60.754.4–56.2

A single generator update already lowers both detector families, which is consistent with the newer generator leaving fewer artifacts and visible flaws. The full results, including those of Gemini, are reported in Appendix F.1 (Table 34).

Thank you again for this valuable comment. We will keep following newly released generators and extending RA-Bench, which also steadily enlarges its scale.

AuthorsOpenReview ↗

Response to Reviewer Tx3r (2/3): W2

W2: The methods are not adequately compared with results on SOTA (state-of-the-art) datasets in the current field, such as GenVidBench and GenVido, thereby weakening the persuasiveness of the experimental conclusions.

Thank you for this suggestion. We have run the comparison accordingly (we take "GenVido" to refer to GenVideo).

Setting. Using the same weights and inference pipeline as in Table 2, we evaluate the seven traditional detectors on GenVideo-Val and on the GenVidBench test set. Each cell reports AUC / T@5% (%), where T@5% denotes TPR at 5% FPR.

DetectorGenVideo-ValGenVidBenchRA-Bench open-sourceRA-Bench closed-source
CNNSpot73.3 / 11.443.4 / 6.147.3 / 5.142.2 / 3.4
NPR41.0 / 2.432.4 / 1.253.8 / 4.650.9 / 4.2
UnivFD79.0 / 33.648.0 / 12.141.7 / 1.039.9 / 1.1
ForgeLens78.4 / 39.864.8 / 29.364.2 / 18.358.7 / 9.7
DeCoF14.0 / 0.153.2 / 6.663.4 / 3.958.9 / 2.0
D397.1 / 86.565.2 / 3.753.8 / 4.535.1 / 5.3
ReStraV75.9 / 41.756.7 / 13.658.1 / 9.654.9 / 5.0
Mean65.5 / 30.852.0 / 10.454.6 / 6.748.7 / 4.4

Analysis. We find that part of a detector's score comes from the video format rather than from authenticity, and this part differs across benchmarks.

  • GenVideo-Val. There is a frame-rate gap between the real and generated sides. The released real videos are stored at 3 fps, whereas the generated videos are 8 fps or higher [1, 2]. This drives the two temporal detectors in opposite directions. D3 exploits the duplicated frames that appear when the 3-fps real videos are resampled [1] and reaches 97.1%. For DeCoF, the 32 frames it samples from span about 10 s in the real videos but at most 4 s in the generated ones, so the real videos show larger inter-frame changes, which is a likely reason for its AUC of 14.0%.
  • GenVidBench. The results are consistent with RA-Bench. The mean AUC is 52.0%, on par with 54.6% and 48.7% on RA-Bench, so these detectors are close to chance on average on both benchmarks.
  • RA-Bench. A controlled experiment confirms the role of format. Unifying all LastMile videos to 640×360 with the content unchanged moves UnivFD from 37.1% to 51.0% AUC. Yet no detector becomes reliable: the mean AUC is 53.0%, and the best T@5% is 20.3% (ForgeLens).

Conclusion. The three benchmarks are complementary.

  • GenVideo and GenVidBench provide million-scale training data and cross-generator test splits.
  • RA-Bench is an evaluation benchmark grounded in real crisis events. It pairs every real anchor with clips from nine recent generators and adds format controls, human judgments, and a dissemination simulation.

The table above and the format control have been added to the revision as Appendix F.2 (Table 35) and Appendix F.3 (Table 36).

References

[1] Michels et al., Dataset Biases and Shortcut Learning in Motion-Based AI-Generated Video Detection, arXiv:2607.00948.

[2] Ben Hayun et al., Training-free Detection of Generated Videos via Spatial-Temporal Likelihoods, arXiv:2603.15026.

AuthorsOpenReview ↗

Response to Reviewer Tx3r (3/3): W3 and W4

W3: There is a lack of performance comparisons with classic detection methods such as Demamba on this dataset, making it insufficient to fully demonstrate the advantages of the proposed dataset.

Thank you for pointing out this representative work. We explain below why DeMamba is not yet included and what the current evaluation already covers.

  • DeMamba. The official weights of DeMamba are not available at present. In the issue tracker of the official repository, the authors state that the weights cannot be released for now due to company policy, and we are not aware of public weights that reproduce the original training setting. To avoid unfair conclusions about the original method, we will evaluate it once the official weights are released.

Nevertheless, our current evaluation still broadly demonstrates the advantages and the difficulty of the proposed benchmark.

  • The detectors evaluated in the paper already cover three families.
    • Seven traditional detectors: four generated-image detectors (CNNSpot, NPR, UnivFD, ForgeLens) and three generated-video detectors (DeCoF, D3, ReStraV).
    • Ten zero-shot MLLM detectors, each under three review settings.
    • Two fine-tuned MLLM detectors, Skyra and BusterX++.
  • Detectors are examined from several angles: cross-generator generalization, generation quality and conditioning, agreement with human judgments, and robustness under social dissemination.

W4: The evaluation and detection methods proposed in the paper focus primarily on specific fully-synthesized video scenarios, lacking sufficient coverage for more challenging real-world complex forgery forms such as localized manipulation, video splicing, or hybrid editing.

Thank you for this suggestion. We agree that these forgery forms are important. Below, we clarify the current scope of RA-Bench, explain why these forms call for a separate setting, and describe our plan to cover them.

  • Current scope. RA-Bench targets clip-level authenticity classification. It focuses on videos generated from a real first frame, which is the misinformation scenario the paper studies. We have stated this scope explicitly in the revision (Appendix F).
  • A different task setting. Localized manipulation, splicing, and hybrid editing require region- or segment-level annotations and localization metrics. They are studied on dedicated datasets such as ForgeryNet [3] and AV-Deepfake1M [4].
  • Ongoing extension. Extending RA-Bench in this direction is our ongoing work. We keep the real-anchor pairing, retain the original video for every edited clip, annotate the manipulated regions or time spans, and evaluate detection and localization separately.

Thank you again for your valuable comments. We hope the responses above address your concerns.

References

[3] He et al., ForgeryNet: A Versatile Benchmark for Comprehensive Forgery Analysis, CVPR 2021.

[4] Cai et al., AV-Deepfake1M: A Large-Scale LLM-Driven Audio-Visual Deepfake Dataset, ACM MM 2024.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to ACs and All Reviewers

Dear ACs and Reviewers,

We sincerely thank the ACs and all reviewers for the time and effort devoted to our submission. We are encouraged that Reviewer Tx3r finds that the paper "constructs a rigorously designed, comprehensive new evaluation benchmark with high academic foresight and practical value". Reviewer c6nE recognizes that it "couples machine detection with human judgment and a dissemination simulation, yielding decision-relevant findings". Reviewer Maag notes that the studied problem "has clear practical relevance" and regards the analysis of Skyra's temporal-label sensitivity as "particularly valuable" and "a meaningful strength of the paper".

We respond to each comment point by point below. The additional experiments and analyses have been incorporated into the revised manuscript as Appendix F, marked in blue.

Best regards,

Authors of Submission 15594

Dear reviewers,

The authors put considerable effort into writing a rebuttal to your points. Could you please read it and respond?

Thank you & kind regards,

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.