acceptodds
Under review as a conference paper at ICLR 2027

OR-Talk: Occlusion-Robust Lip Synchronization via Selective Appearance Transport

Abstract

Audio-driven lip synchronization modifies mouth motion according to a target speech signal while preserving identity and head motion. Under facial occlusion, existing methods often distort or remove foreground occluders, since preserving occluder appearance conflicts with audio‑driven control of mouth shape. We propose OR-Talk, which formulates occluder recovery as an appearance transport problem. Based on this formulation, we introduce Occlusion Adaptive Transport (OAT) to selectively transfer the appearance cues needed for occluder reconstruction from the identity features to the decoder. To avoid leaking mouth information during this process, we construct paired identity conditions with matched head poses but different mouth shapes. This design preserves occluder appearance while keeping the mouth shape controlled by the input audio. Experiments show competitive performance on standard benchmarks and substantially improved robustness to facial occlusion. Under 50–100% mouth occlusion, OR-Talk improves PSNR by 6.65 dB and reduces OCC-E by 51.6% compared with X-Dub.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Lip Synchronization, Talking Face Generation, Visual Dubbing, Facial Occlusion

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Jj38OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses occlusion robustness in lip synchronization by recasting occluder recovery as selective appearance transport. The proposed OAT mechanism, trained with pose-aligned identity pairs, selectively transfers occluder features without leaking reference mouth shapes. Experiments on standard and occlusion-specific benchmarks confirm substantial gains in occluder preservation while maintaining competitive lip-sync quality.

Critical strengths
  • The manuscript is well-written and the motivation is clear.

  • The Occlusion Adaptive Transport (OAT) and pose-aligned identity conditions effectively prevent ground-truth mouth leakage during training.

  • The novel occlusion-specific protocol and the OCC-E metric ensure fair assessments without target data leakage.

  • The experimental results and the supplementary demo exhibit a substantial and convincing margin of improvement over SOTA baselines.

Critical weaknesses
  • The framework relies on a two-stage pipeline where Stage-II supervision depends on pseudo identity images generated by a frozen Stage-I model. This design tightly couples the final occlusion robustness to Stage-I's disentanglement capacity, making the system vulnerable to error propagation and artifact accumulation.

  • While the proposed OCC-E metric effectively quantifies frame-wise, pixel-level reconstruction accuracy within the occluded region, it is inherently a spatial measure. It lacks temporal awareness and fails to capture potential temporal flickering or structural inconsistencies of the occluder across video sequences.

Questions for authors

How does OR-Talk quantitatively compare against recent methods such as KeySync? Without these comparisons, the claim of SOTA robustness remains unsubstantiated.

1 comment

My concerns have been addressed. I appreciate the author adding a further comparison with KeySync. The author provided a breakdown of frame processing times in comparison to KeySync, which makes the tool seem promising. I will keep my score.

Reviewer_L9cJOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposed OR-Talk, an audio-driven lip-synchronization method designed to be robust against foreground occlusions. The authors introduce occlusion-aware training with an occlusion augmentation strategy and an Occlusion Adaptive Transport (OAT) module to transfer identity and occluder appearance features to the decoder. The method uses two-stage training and reports strong quantitative results on standard lip-synchronization datasets and an occlusion-specific benchmark.

Critical strengths
  1. The occlusion augmentation strategy is interesting and effective as demonstrated in the ablations, even when applied to other methods.
  2. The quantitative results are strong across visual quality, lip-synchronization, and occlusion preservation.
Critical weaknesses
  1. OAT is essentially deformable cross-attention with mask guided residual feature fusion. The proposed operation has limited novelty apart from its application in a new task of lip-sync.
  2. The two stage training and pose-aligned identity are not sufficiently motivated. The authors argue that transferring identity features can leak reference mouth information, however, Stage I already receives identity features containing mouth information. It is unclear why this leakage risk is specific to Stage II. Furthermore, it is unclear why the Stage I generated pose-aligned identity is required instead of a simpler reference frame strategy, like pose based reference selection. Although the ablations demonstrate the benefit of occlusion augmentation and OAT, they do not isolate the contribution of the synthetic identity construction. Finally, why do you need 2 stages - couldn’t the authors jointly train everything in the first stage with the occlusion training and OAT, using a simpler reference frame selection strategy?
  3. OmniSync (NeurIPS 2025) is a relevant occlusion-robust lip-sync method but is not included in comparison. Including this baseline would help establish the proposed method’s competitiveness.
  4. Qualitative synchronization results are difficult to assess. In the supplementary video, the HDTF and LRS2 examples appear poorly synchronized across all the methods, however this is not the case for VoxCeleb2. Furthermore, the occlusion examples have the mouth often obscured, making it difficult to examine whether the lips are accurately driven. Additional qualitative examples would help evaluate synchronization quality.
Questions for authors

Please see the weakness section.

Reviewer_pjvpOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents an occlusion robust lipsync method. It is known that most lipsync methods can lead to distortions if facial occlusions are present and this paper introduces occlusion adaptive transport which selectively transfers the appearance cues and prevents leakage of the mouth shape. The proposed approach is evaluated on multiple datasets and comparisons with other SOTA methods are presented. In addition, some ablation studies, showing the effectiveness of the proposed methods, are conducted.

Critical strengths

The paper addresses the issue of mouth distortions when lipsync models are presented with occlusions. This is a useful contribution in an underexplored area.

Critical weaknesses

Major Weaknesses

  • There is no user study included in the paper. The paper includes several metrics, however these are not always highly correlated with human perception. A user study is essential in video generation works as quality cannot be judged only by automated metrics.
  • There is no comparison with KeySync (KeySync: A Robust Approach for Leakage-free Lip Synchronization in High Resolution), which is one of the SOTA methods for occlusion-robust lipsync.
  • There is no ablation study on the losses used. There are 5 losses used in the paper and an ablation study removing one loss at a time is needed to reveal if each loss is necessary.
  • Evaluation is rather weak. Just 30 videos from each dataset are added in the test set. It would be better if many more videos were part of the test set. It would make the results more convincing.
  • The occlusions used are not realistic. Adding random objects in front of the mouth results in non realistic scenarios. It would be good if the authors focused on more realistic occlusions. As a minimum, this should be mentioned as a limitation of the work.

Other Weaknesses

  • Few and short examples are included in the video supplementary material. Including more examples would be desirable.
  • It should be explained in the captions why some numbers are bold or underlined in the tables. I guess they correspond to the best and second best performance, respectively.
  • Section 3.2, It is not clear why the second encoder is called head motion, it probably captures head pose related information so a better name would be head pose encoder.
  • There is very little information provided about the encoders and the decoder used. A figure or a table in the appendix is needed explaining the architectures in detail.
  • Section 4.1, it is not clear how the loss weights are selected.
  • A figure in the appendix, showing examples of varying degrees of overlap would be useful so a read can better understand what the different occlusions percentages mean.
Questions for authors

Please check the major weaknesses section above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.