- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work introduces a new MLLM-based audio-visual method for detecting and localizing deepfake manipulations. The method is equipped with auxiliary components (the representation extractors, shift-alignment network and Boundary-AEP module) that provide explicit evidences extracted from audio and visual streams that are further used by the MLLM to predict evidence-grounded decisions and diagnosis. Furthermore, the authors introduce two key components: BLA-AMR, which regulates the learning process of the Boundary-AEP, balancing the contribution of each modality and Safe-AMR, which is responsible for alleviating the reliance on cross-modal shortcuts (audio dataset-specific or audio-visual disagreement shortcuts). The evidence is used by a split-modal MLLM trained using modality-specific LoRA that analyzes it together with the input audio-visual sample to produce localization predictions and corresponding explanations. The authors evaluate their method against a set of deepfake temporal forgery localization methods on two widely used datasets: LAV-DF and AV-Deepfake1M, achieving state-of-the-art performance.
Critical strengths
- This work addresses two important issues:
- inconsistency methods are limited to detecting audio-visual disagreements, yielding predicted regions that cannot be categorized as visual-only, audio-only or audio-visual manipulated since this information is missing. EviPatrol has specialized modules that regulates the model such that it is able to correctly predict visual-only, audio-only and audio-visual manipulated regions.
- exiting approaches learn implicit forensic cues in the latent space through supervised learning, transforming them into black boxes. As a consequence, classic methods can provide localized predictions but without a clear evidence (especially no textual explanations). This raises interpretability concerns, a common issue of deepfake detection. On the other hand, EviPatrol is able to predict a full diagnosis of the detection decision.
-
The method's performance is supported by experimental results, achieving substantial improvements over state-of-the-art methods especially under a strict localization setup (). Moreover the ablations prove the capability of the full framework.
-
While some information is missing, the work is generally well structured, clear and easy to follow. The contributions are clearly stated and the motivation is well justified.
Critical weaknesses
Anchors:
- The granularity of anchors can determine the detection’s quality. As such, the choice of 0.5 seconds/ a maximum of 32 anchors should be ablated.
- What happens when a video is substantially larger than samples from AV-Deepfake1M/LAV-DF (especially for in-the-wild data which tend to contain significantly longer videos)? The anchors' granularity decreases which might lead to a quality degradation.
Boundary-AEP:
- This component contains a wide range of modules. While theoretical justifications are provided for these choices, a more granular ablation is required to confirm empirically the necessity of each module.
- It is not clear on what data is the reconstruction module from Boundary-AEP trained on. Is it trained on real-data only or also on manipulated data? For the latter, the reconstruction module will learn to reconstruct manipulated regions as well as normal regions, which defeats the purpose of this module.
Safe-AMR:
- For the paired-state discrepancies computation, do the two corresponding videos contain the same semantic input? Otherwise the difference between the two normalized decision losses do not reflect a discrepancy when audio is manipulated, but rather different sample difficulties. The authors should specify this.
- Why is Safe-AMR evaluating only in one direction? Visual evidence that influences audio decisions is also possible in theory. The authors should provide evidence that supports their hypothesis.
- In the main paper the formulae for and are presented. Then, in appendix D.2 the authors replace the computation with . Furthermore, the tolerance margin is no longer present in the final formula. This is a writing consistency issue that makes the Safe-AMR methodology hard to follow and should be addressed.
- the value of is not specified and it is highly dependent on the output formatting.
Evaluation:
- Authors should specify what baselines are retrained/finetuned and for what baselines the trained checkpoints were utilized.
- It is unclear if the video-only methods are evaluated against the video-only groundtruth, specifically on AV-Deepfake1M. If the methods are evaluated on the union ground-truth, then the evaluation is unfair. If not, the evaluation is not comparable. The authors should specify this.
- A comparison with another MLLM-based temporal forgery localization method would have improved the baseline comparison greatly. For example, using Omni-Fake-R1, a method already cited by the authors.
Minor:
- mDPO is wrongly cited.
Questions for authors
- is computed on training or validation splits? On a mini-batch or averaged across the full split?
- What happens if the split between two anchors is done during a phoneme or viseme? Wouldn’t this lead to a quality degradation?
2 comments
Dear authors,
Thank you for your response. While some of the concerns were (partially) addressed, I would have appreciated more empirical proofs (e.g. performance degradation based on video lengths, ablation on Boundary-AEP components, comparison against MLLM methods) in order to further prove the applicability and usefulness of EviPatrol. Furthermore, several concerns are still not addressed. As a result, I will keep my Weak Rejection rating.
Response to Reviewer 7jD5
Dear Reviewer,
Thank you for your comments. We address the main points below.
W1 / Q2. Anchor resolution, long videos, and phoneme boundaries. Anchors index evidence on a nominal 0.5-second grid; the MLLM predicts continuous timestamps rather than anchor-quantized boundaries. When more than 32 anchors are needed, Appendix B.1 uses 32 duration-adaptive windows covering the full timeline without truncation. Wider windows may weaken brief-anomaly representation; full coverage does not establish robustness at arbitrary video lengths.
Dense features are extracted before pooling. Intra-anchor transitions and cross-anchor temporal modeling retain context, so a cut within a phoneme or viseme does not directly discard that information.
W3. Semantic matching in Safe-AMR. Safe-AMR uses two kinds of pairing. RF/FF and RR/FR compare state groups with the same visual label but different audio states; they do not require semantically matched source content. Their loss gap is a training-monitoring signal, not by itself a causal effect of changing audio.
The normal and audio-withheld branches, in contrast, form a same-example intervention pair: the visual input and visual target remain fixed while the audio condition changes. This avoids the content and difficulty differences inherent in comparing two different videos. Safe-AMR uses both state-group statistics and same-example decision sensitivity, rather than relying only on the former.
W5 / Q1. Formula consistency and training statistics. The appendix equation omits the margin from the main-text definition. The correct expression averages the two margin-thresholded discrepancies:
Δt=21(Δ+,t+Δ−,t)
The tolerance margin affects the shortcut score and counterfactual sampling probability through these discrepancies. It does not appear again in the final loss.
These statistics are updated online from training-batch observations, synchronized across devices, using state-specific EMAs. They do not use validation data or full-split recomputation. Controls are read before the current observations update the controller.
W6. Ground-truth alignment and comparability. V-AP and U-AP measure different tasks. A like-for-like V-AP comparison uses visual ground truth for both video-only baselines and EviPatrol. A U-AP comparison uses union ground truth for every method. A video-only result on the union task is a restricted-input reference, not an input-matched audio-visual comparison.
W7. Comparison with Omni-Fake-R1. Its public audio-visual talking-head branch predicts global authenticity rather than modality-specific temporal intervals. Other branches provide audio temporal localization and video spatial localization. The public evaluator reports IoU/F1, not temporal AP.
Its published results therefore cannot be inserted directly into our same-protocol comparison. A task-adapted baseline would require extending audio-visual temporal outputs and adapting supervision, rewards, and evaluation.
Minor. mDPO citation. The correct reference is Wang et al. (2024), mDPO: Conditional Preference Optimization for Multimodal Large Language Models.