acceptodds
Under review as a conference paper at ICLR 2027

Beyond Lip Sync: Reference-Grounded Oral Refinement for Audio-Driven Portrait Animation

Abstract

We present RGOR (Reference-Grounded Oral Refinement), an audio-driven lip-sync framework that renders the mouth of the specific person being dubbed rather than a generic one. Existing lip-sync systems follow the audio closely and keep the face recognizable, yet the mouth they render is an average mouth: the shape and texture of the lips, the arrangement of the teeth, and how much of them shows as the mouth opens are not that person's. The problem persists because nothing in current training or evaluation asks for the person's own mouth: perceptual losses accept any plausible mouth, face identity is carried mostly by the skin around it, and the released inference code of inpainting systems uses the unmasked target frame as the reference, which hides the gap. To address this, RGOR conditions every generated frame on frames from separate enrollment recordings of the same person and on HD patches of the mouth that bypass the VAE, and trains the generator against a paired judge that compares each rendered mouth with the person's reference and learns to reject a realistic mouth of someone else. We further build an evaluation protocol and use it to compare open-source and commercial lip-sync systems on held-out identities. Experiments show that RGOR achieves the best or second-best result on most metrics, and preserves the person's own lip and dental detail while keeping synchronization and the rest of the face intact.

Primary area:
generative models
Keywords:
lip sync, visual dubbing, audio-driven portrait animation, diffusion models, identity preservation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_VupROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents RGOR (Reference-Grounded Oral Refinement), a framework that improves the identity consistency of generated lips and teeth through reference-guided generation. The method introduces scattered enrollment references and high-resolution mouth patches to preserve fine details. In addition, the authors introduce a reference-contrastive paired loss and a shape loss to better preserve person-specific mouth appearance.

Critical strengths
  1. The paper has conducted sufficient experiments and human study, and RGOR has good numerical results on benchmarks.
  2. The visualization cases show improvements in lip and dental details.
Critical weaknesses
  1. The comparison may be not fair. It is unclear whether the other baselines were trained on the same dataset.
  2. I am concerned about the practicality of this work. RGOR performs worse than LatentSync-1.6 on the ood HDTF dataset despite additional training. Moreover, existing methods like InfiniteTalk seem to already handle lip identity consistency well.
Questions for authors
  1. See weekness.
  2. Could the authors provide a comparison with InfiniteTalk?
Reviewer_tRbNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents RGOR, an extension of LatentSync for preserving person-specific lip and dental details in audio-driven visual dubbing. It introduces scattered references from separate recordings, an HD mouth-patch branch that bypasses VAE encoding, a reference-conditioned paired discriminator, and a segmentation-based shape loss. Evaluation includes an identity-disjoint NeRSemble benchmark, component ablations, a human study, and an out-of-distribution test on HDTF. RGOR improves perceptual fidelity on NeRSemble, although its advantage does not consistently transfer to HDTF.

Critical strengths
  1. A meaningful fidelity problem with encouraging results. Preserving individual lip and dental characteristics is practically important and is not adequately captured by synchronization or whole-face identity metrics alone. The improvements on NeRSemble are substantial: oral DISTS decreases from 0.234 for cross-reference LatentSync to 0.163, and the human study also favors RGOR.
  2. Useful scrutiny of evaluation protocols. The cross-clip enrollment/query split and the comparison against same-frame references expose a source of optimistic paired reconstruction scores. This analysis is valuable beyond the proposed model.
Critical weaknesses
  1. The methodological novelty remains limited and insufficiently differentiated. Mouth averaging and identity-specific texture preservation have already been studied by UnAvgLip, IPTalker, and EfficientSync. Although these works are discussed, they are not evaluated. The proposed components largely adapt established reference-conditioning and adversarial-learning mechanisms. In particular, the HD branch needs clearer differentiation from conventional reference-feature injection, while the paired discriminator is not independently shown to measure oral identity rather than realism or mouth geometry.
  2. The experiments do not sufficiently isolate the contribution or establish robust generalization. RGOR is fine-tuned on NeRSemble, but there is no parent-model control fine-tuned on the same data with a matched budget. On HDTF, RGOR performs worse than LatentSync on several central fidelity and synchronization metrics. The ablations provide useful evidence, but lack repeated-run uncertainty estimates; the HD branch’s DISTS improvement is relatively small, from 0.165 to 0.161. These gaps limit confidence that the reported gains reflect a stable methodological advance.
Questions for authors
  1. Can you provide a LatentSync control fine-tuned on the same NeRSemble data with matched training settings, together with comparisons against the closest identity-preserving methods, particularly UnAvgLip and IPTalker?
  2. Can you isolate the contribution of real mismatched-identity negatives by comparing the paired judge with an otherwise identical discriminator without those negatives? An independent mouth-identity evaluation controlling for mouth opening would help establish what the judge learns.
  3. Does the HD branch outperform an otherwise matched branch using VAE-encoded mouth patches? Please report repeated-run variability for this comparison and clarify the branch architecture, loss weights, and judge-pretraining data and identity split.
Reviewer_jUfdOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses person-specific oral identity preservation in audio-driven visual dubbing, an aspect largely overlooked by conventional lip-sync methods and evaluation metrics. The authors propose Reference-Grounded Oral Refinement (RGOR), built upon LatentSync-1.6, to preserve identity-specific details such as lip appearance, tooth structure, and mouth-opening characteristics. RGOR incorporates scattered cross-clip enrollment references, high-resolution mouth patches that bypass VAE compression, a reference-contrastive paired judge for oral identity supervision, and a shape loss for mouth geometry. The paper also introduces a cross-clip evaluation protocol to avoid reference leakage in paired reconstruction. Experiments on an identity-disjoint NeRSemble benchmark, including comparisons with several open-source and commercial systems, show that RGOR improves oral perceptual similarity, texture preservation, and mouth-opening fidelity, while maintaining competitive lip synchronization and consistency outside the edited mouth region.

Critical strengths
  1. The paper addresses an important problem in lip synchronization: whether the generated mouth actually belongs to the person in the original video, rather than looking like a generic mouth.

  2. The paper is well-written and easy to follow.

  3. The experiments are comprehensive, with comparisons against a wide range of open-source and commercial models.

Critical weaknesses
  1. The quality of the generated mouth movements seems somewhat limited. From the supplementary materials, the generated lip movements do not seem very natural in some cases. In particular, the mouth often opens too little, resulting in a noticeable mismatch with the audio.

  2. I have concerns about the generalization ability of the proposed method. The training dataset contains a relatively limited number of identities and scene variations, and the supplementary results mainly show simple, frontal talking-head videos captured in controlled environments. More specifically:

    (a) How well does the proposed discriminator generalize when trained on a relatively small number of identities? Is it only effective for the identities seen during training, or can it also distinguish the mouths of different unseen identities, especially on out-of-domain data? It would be helpful to include an experiment evaluating the discriminator's performance on unseen identities and datasets.

    (b) This concern about the discriminator's generalization also raises questions about the generalization of the trained generator. Can the model still effectively preserve person-specific dental identity on out-of-domain videos?

  3. I am not fully convinced by the motivation regarding reference leakage in previous methods. The authors argue that using the corresponding video frame as a reference during inference hides the problem of inaccurate mouth identity. However, using a reference frame with a similar background, head pose, and other visual properties is naturally beneficial for generation quality. This is also a reasonable inference setting in practice, and it does not necessarily introduce lip-motion leakage. Therefore, I would not consider the performance gains from such references to be simply hiding the underlying problem.

  4. Based on experience, when LatentSync uses reference frames that are not aligned with the target frames (i.e., not the corresponding frames from the input video), the generated results can suffer from severe temporal jitter and perform poorly under occlusions or side-view poses. Since the proposed method is built on LatentSync but uses reference frames sampled from a separate enrollment bank, does it also suffer from these issues? The current qualitative results mainly focus on simple frontal-face cases, making it difficult to assess robustness under more challenging conditions.

  5. The reference selection process during inference is not entirely clear. For unseen identities, does the method require manually selecting suitable reference frames, or does it rely on a rule-based selection strategy? If the available enrollment video is short or of limited quality, could the method fail because it cannot obtain sufficiently informative reference frames?

Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.