acceptodds
Withdrawn from ICLR 2027

MoCouple: Learning Fine-Grained Motion Coupling for Generalizable Talking-Face Forgery Detection

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Advances in talking-face generation (TFG) enable increasingly convincing audiovisual impersonation, heightening the risks of identity misuse and privacy violations. Visual artifact analysis and audiovisual synchronization analysis constitute two dominant paradigms in current detection methods, with the latter primarily focusing on speech and lip dynamics. However, generator-specific artifacts can limit transfer to unseen synthesis methods, while accurate lip synchronization can render audio–lip mismatch less discriminative. Lip-centered analysis can also overlook subtle inconsistencies across facial regions and become vulnerable when non-frontal views obscure mouth dynamics. Although recent TFG methods increasingly capture audio–motion correspondences, aligning each motion component with audio alone does not ensure natural coordination across motions. Motivated by this, we propose MoCouple to detect subtle cross-component inconsistencies by learning the joint coordination of lip articulation, facial deformation, head movement, and speech from genuine videos. Specifically, masked latent reconstruction on real videos learns how lip articulation varies jointly with speech, facial deformation, and head motion, while window-level motion statistics capture weaker dependencies between audio and motion. Fusion guided by quality and evidence then combines these relation sensitive features with visual forensic cues to detect forgeries with plausible local motions but inconsistent joint dynamics. Experimental results demonstrate that, in addition to maintaining strong performance on the original training dataset, our method achieves significant out-of-distribution generalization capabilities that surpass existing methods. This work positions fine-grained motion coupling as a complementary authenticity criterion beyond visual fidelity and audio–lip alignment. The same perspective may inform the evaluation and training of TFG models, encouraging coherent joint dynamics rather than merely plausible movements of individual regions.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Deepfake Detection, Audio-Visual Deepfake Detection, Multimodal Representation Learning, Cross-Modal Consistency, Audio-Visual Alignment

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_EREkOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes MoCouple, a framework designed to enhance the generalizability of audio-visual deepfake detection by learning fine-grained motion coupling.

Critical strengths

The motivation is good, with the authors pointing out that relying solely on local audio-lip synchronization is inadequate. Natural speaking inherently demands complex joint coordination among speech, facial expressions, and head movements.

Critical weaknesses
  • The validation experiments in Section 3 fall short of fully supporting the claim of successfully extracting fine-grained joint features. Specifically, this section relies merely on distribution visualization and the MMD metric, lacking independent validation and ablation studies to demonstrate how each pre-training objective effectively captures specific motion dependencies.
  • Moreover, the proposed cross-modal feature fusion and evidence fusion designs lack novelty.
  • This paper also suffers from poor readability. Figure 1 is blurry, and the caption for Table 1 should be placed consistently above the table.
Questions for authors

Robustness evaluations under complex visual conditions are absent. Could the authors provide additional robustness validation results for MoCouple in scenarios involving facial occlusion and varying illumination?

Reviewer_PevNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MoCouple for talking-face forgery detection. The motivation is that convincing lip synchronisation does not necessarily imply realistic coordination between speech, lip movement, facial expression and head motion. The method models audio–vocal, audio–motion and visual-motion relationships using BRAVEn and VideoMAE features. This relation stream is first pretrained on authentic videos through masked reconstruction and motion-statistics prediction, then trained for forgery detection. A separate forensic stream combines CLIP features with spatial and temporal artefact cues. The streams are combined using feature fusion and estimates of their confidence and disagreement. Experiments train on MMDF and evaluate unseen generators and transfer to THB and FakeAVCeleb, with an additional AVLips evaluation in the appendix. The method obtains the best reported dataset-level average in Table 1, with particularly strong results on THB.

Critical strengths

I like the motivation for looking beyond audio–lip synchronisation, and the analysis in Fig. 1 gives a concrete reason to explore this. The differences between real and generated videos are more apparent in the broader motion-relation representations than in the audio–vocal representation. Including MMD measurements in the original feature spaces is also useful alongside the UMAP plots. This makes the motivation more convincing than only relying on a few qualitative examples.

The main results are competitive. MoCouple achieves the highest dataset-level average in Table 1, and the THB result of 95.76 AUC is substantially above the other methods listed. The relation stream alone also performs well on THB and FakeAVCeleb. These results suggest that the relation stream learns useful information for transfer on some datasets, which is an important part of the paper's contribution.

Critical weaknesses

My main concern is that there are not enough ablations to understand where the performance is coming from. The method combines several strong pretrained models, additional pretraining on real videos, three relation branches, several losses and quite elaborate fusion. Table 2 only compares the two complete streams and their combination. This shows that the streams can be complementary, but does not tell us how much is gained from authentic-motion pretraining, from modelling the additional motion relationships, or from the disagreement and evidential fusion mechanisms. In particular, a simpler detector using the same pretrained features would be an important comparison. Without these experiments, I find it difficult to judge how much the results support the proposed fine-grained motion coupling itself.

The generalisation results are not consistently strong. The average in Table 1 is good, but there are sizeable gaps to other methods on some targets: 82.16 AUC on HunyuanAvatar compared with 99.94 for DiCoME, and 85.23 on FakeAVCeleb compared with 99.69 for X-AVDT. More concerning is the 56.01 AUC on AVLips in Appendix D. The authors acknowledge this failure and discuss lower visual quality and spatial jitter as possible explanations, but the analysis is mainly qualitative. Given that generalisation is a main claim, I would like a better experimental understanding of this failure. Fusion also reduces AVLips performance from 58.16 for the relation stream to 56.01, which raises a question about how well the proposed reliability estimates handle a degraded stream.

Questions for authors

What happens when authentic-motion pretraining is removed? Keeping the pretrained encoders and supervised training setup fixed, how much is gained by adding the audio–motion and visual-motion branches to an audio–vocal-only model?

How does the proposed fusion compare with simple score averaging or feature concatenation?

Could you clarify how the expression and head-motion representations (H_e) and (H_h) in Eq. (1) are obtained from the shared VideoMAE encoder? What encourages them to represent these different motion components?

Reviewer_UfN4OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MoCouple for talking face forgery detection by modeling relationships beyond audio-lip synchronization. A relation stream learns representations through real-video pretraining and supervised forgery detection, while a forensic stream captures complementary visual features. Experiments report good results on the primary cross-generator and cross-dataset evaluations, alongside a challenging transfer setting where performance degrades substantially.

Critical strengths
  1. Modeling broader facial coordination could provide useful evidence when generated videos preserve plausible local articulation.
  2. The relation stream performs strongly on the primary external benchmarks, while combining it with forensic features improves other evaluated settings.
Critical weaknesses
  1. The expression and head-motion representations are derived from a shared video encoder, but their semantic roles are not validated. Distributional differences in these features do not establish whether the model captures abnormal motion coordination or general appearance differences.
  2. The paper lacks a standard feature-fusion baseline using the same backbone and pretraining setup. It is unclear how much of the improvement comes from fine-grained relation modeling and evidence-aware fusion.
  3. Directly relevant methods, such as AVH-Align, AVFF, and SpeechForensics, are missing. It is also unclear whether the imported X-AVDT results were evaluated on exactly the same video subset, making the overall ranking less convincing.
  4. Performance on AVLips is close to random, and fusion performs worse than the relation branch alone. Although the authors acknowledge this failure, the explanation based on preprocessing jitter lacks controlled validation, leaving the applicability of the method unclear.
Questions for authors
  1. How are the expression and head-motion representations constructed? What evidence shows that they capture the intended motions and that detection relies on the correspondence between them?
  2. How does standard feature fusion compare with the full method under the same backbone and real-video pretraining setup?
  3. Could the authors clarify the baseline selection criteria and provide key comparisons under the same evaluation protocol? Were the imported X-AVDT results obtained using exactly the same evaluation videos?
  4. Are there controlled experiments showing that preprocessing jitter is the main cause of the performance drop on AVLips?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.