acceptodds
Withdrawn from ICLR 2027

DUET: Audio-Driven Human Video Synthesis with Voice-Face Dual Harmonization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Existing audio-driven human video generation predominantly focuses on lip synchronization, yet often overlooks the timbre mismatch between the driving voice and the depicted speaker. We propose DUET, a voice-face dual-conditioned talking video generation model that animates the face with lips synchronized with the driving audio while adaptively updating the audio timbre to match the speaker appearances. We formulate this task within a joint audio-video generation framework: the driving utterance provides linguistic and temporal cues, while the conditional image determines the vocal identity of the generated speech. We incorporate a Whisper-based audio injection branch into a pretrained joint audio-video generative model and train it on timbre-perturbed audiovisual pairs to facilitate both lip synchronization and timbre reasoning through cross-modal learning. For multi-person conversational scenes, we introduce a spatio-temporal masking mechanism that associates each audio condition with its designated face region and speaking interval. On the speaker-identity-disjoint EMTD benchmark, our method obtains the best or tied-best scores on the evaluated voice-conversion metrics. On EMTD and InterActEyes, it maintains competitive visual quality and lip synchronization. In a perceptual comparison with a DreamVoice+MultiTalk cascade on EMTD, participants prefer our method in 64.29% of responses for timbre–identity consistency and 53.97% for overall audiovisual coherence.

Primary area:
generative models
Keywords:
video generation, multimodal generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_MCQNOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces DUET, an audio-driven human video generation model that regenerates speech instead of preserving the original speaker’s voice. The approach enhances the pretrained Ovi backbone by incorporating Whisper-based audio conditioning and training on audiovisual pairs with altered timbre. For two-person conversations, a spatio-temporal attention mask links generated speech segments to specific visual regions. Experiments assess voice conversion, video quality, and synchronization on EMTD and InterActEyes datasets.

Critical strengths
  • The task is well motivated. Lip synchronization and voice-face compatibility are distinct requirements. The paper addresses a valuable scenario where the driving utterance determines the content without necessarily specifying the speaker's voice.
  • The voice-conversion results are encouraging under the stated protocol.
  • The conditioning ablation supports the choice of Whisper features. The results shows substantially lower WER and CER than all three VAE-based alternatives.
Critical weaknesses
  • The methodological contribution beyond the pretrained backbone is limited. DUET inherits its joint audio-video generator and flow-matching objective from Ovi, with the main additions being Whisper cross-attention, SeedVC-based timbre perturbation, and masked cross-modal interaction. These are reasonable adaptations, but the method is largely a new conditioning and training setup for an existing generator. My concern is not architectural simplicity, but that the paper does not demonstrate a clear technical advance beyond adapting the backbone to this task.
  • The evaluation does not demonstrate that vocal identity is inferred from the image, as confounding factors remain. Further testing with independent recordings and models without visual conditioning is needed to attribute speaker similarity gains to visual identity inference.
  • Perceptual compatibility is not adequately distinguished from individual vocal identity. The paper explicitly aims to infer the depicted person’s vocal identity, whereas age and gender agreement only assess demographic compatibility. The user study evaluates whether the generated voice matches the visible speaker without documenting the use of the actual target voice as a reference, so it cannot independently establish person-specific identity recovery.
  • The attention formulation omits the handling of silent audio tokens. Under Eqs. (4) to (7), an audio token outside all speaking intervals has no permitted video keys, producing an attention row containing only negative-infinity logits when the transposed mask is applied. Ordinary softmax is undefined for such a row. The implementation may handle fully masked queries safely, but this behavior should be specified.
Questions for authors
  • How much of the SECS improvement remains without visual conditioning, and does the improvement persist with independently recorded driving speakers?
  • A portrait-swap experiment with fixed audio and text conditions would help determine whether the output accurately reflects the intended individual’s identity.
  • Can the authors report per-speaker voice similarity, assignment accuracy, and unintended mouth motion during the other person’s speaking turn? These measurements would address the claimed binding capability more directly than the current metrics.
Reviewer_3c5jOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a method that generates new talking videos from an audio input and a reference face image, with results that demonstrate harmonization between the audio timbre and speaker appearance. Say, input a man's audio as the driving source to animate a woman's portrait; the synthesized video would be a talking video in which the corresponding audio demonstrates the matched woman like timbre. This setup was not widely studied before; the authors argue that a straightforward baseline, which is converting the voice first and then running the traditional talking head generation, is suboptimal due to the accumulated errors. The proposed approach is a joint learning framework that tries to implicitly disentangle the timbre from the rest of the audio representation and model the conditional distribution based on the input visual identity. The approach itself is reasonable; the results in terms of some numeric metrics look strong compared with a straightforward baseline.

Critical strengths

The problem itself is an interesting formulation; the overall methodology is straightforward and seems to be easy to implement. The experimental section is rich, and some metrics are strong compared with the baseline. The supplemental materials are very helpful for understanding the whole problem and the resulting quality.

Critical weaknesses

I have two main concerns

  1. I understand there is prior work that studies the correlations between audio timbre and facial appearance/identity; however, the conclusion seems to be that their correlation is weak: two people with drastically different visual appearance could have very similar voice timbre. In other words, even if the proposed approach can separate the input audio's original timbre, the resulting video's timbre would have many possibilities; therefore, how to faithfully evaluate the matchness of the result becomes very subjective and hard to measure numerically. Is the method basically just doing random sampling as long as it does not violate the very basic mismatching (i. e., old man VS young kid, girl VS boy)? Also, some results are just not matched well, i.e., in case 5 of the supplemental results, the resulting timbre does not match the boy clearly.

  2. The resulting quality does not pass my subjective quality bar, especially the preservation of the input facial identity, as can be easily seen from the supplemental results, even though it's hard to tell those are cherry-picked ones. The baseline approach is more stable in terms of preserving the identity. Despite the provided user studies, I don't feel the quality is great enough to fully support this paper's claim.

Questions for authors

Would adding more training data improve the visual quality?

What caused the failure to preserve facial identity?

Reviewer_FRUtOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • DUET jointly generates speech and talking videos from driving audio and a reference image.

  • Built upon Ovi, it introduces Whisper-based audio conditioning and timbre-perturbed training to preserve speech content while adapting vocal characteristics to the depicted person.

  • Experiments evaluate voice conversion, video quality, synchronization, and perceptual preference.

Critical strengths
  • Jointly addressing voice–appearance compatibility and audio-driven animation is a useful extension of conventional talking-video generation.

  • Clear improvements under the evaluated protocol. In Table 1, DUET improves SECS from 0.1721 to 0.2946 and reduces WER from 0.1638 to 0.0943 relative to DreamVoice.

Critical weaknesses
  • The methodological novelty is limited. DUET adopts Ovi’s architecture and relies mainly on bidirectional cross-modal attention. Its main extensions are timbre-perturbed training and spatio-temporal masking. However, together with the cross-attention design, offer limited novelty. More importantly, the existing ablation studies do not clearly isolate the contribution of timbre perturbation or quantify the gains of the masking mechanism over similar region-based binding strategies under controlled conditions.

  • The authors’ performance in scenarios involving two speakers is somewhat lacking; only two such demonstration cases are provided. Furthermore, in the second case—specifically under the "Ours1" configuration—audible artifacts (noise) appear during pauses, which sounds unnatural. The authors should clarify whether this issue stems from inherent limitations in the spatiotemporal masking approach. Additionally, they should consider whether further validation in multi-speaker scenarios (large than 2 person) is necessary to demonstrate the effectiveness of this spatiotemporal masking technique.

  • The SECS scores reported in the paper are based solely on an ECAPA-TDNN evaluator. However, ECAPA-TDNN is no longer the latest or most commonly used speaker encoder. The authors should evaluate speaker similarity using a WavLM-based system or a more recent speaker verification model.

  • The binding of generated voices to individual speakers in two-person scenes lacks direct evaluation. The two-person experiments in Tables 2 and 6 report only FID, FVD, and Sync-C, without per-speaker identity similarity, voice-assignment accuracy, or misbinding rates.

Questions for authors
  • Equation (4) selects audio tokens by speaker region and speaking interval, but does not explicitly constrain the video-token timestamp. Is temporal alignment handled entirely through positional encoding and learned attention, or is additional temporal gating applied? Please also clarify how fully masked audio-query rows are handled during silence.

  • The authors should also report LSE-D results in Table 2 and Table 6.

  • The authors should also explain why their method obtains lower synchronization scores than MultiTalk and AnyTalker in two-person scenes, and clarify whether this is due to inaccuracies in SyncNet or limitations of the proposed method. I would be glad to raise my score if the authors convincingly address these concerns and the identified weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.