- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduces DUET, an audio-driven human video generation model that regenerates speech instead of preserving the original speaker’s voice. The approach enhances the pretrained Ovi backbone by incorporating Whisper-based audio conditioning and training on audiovisual pairs with altered timbre. For two-person conversations, a spatio-temporal attention mask links generated speech segments to specific visual regions. Experiments assess voice conversion, video quality, and synchronization on EMTD and InterActEyes datasets.
Critical strengths
- The task is well motivated. Lip synchronization and voice-face compatibility are distinct requirements. The paper addresses a valuable scenario where the driving utterance determines the content without necessarily specifying the speaker's voice.
- The voice-conversion results are encouraging under the stated protocol.
- The conditioning ablation supports the choice of Whisper features. The results shows substantially lower WER and CER than all three VAE-based alternatives.
Critical weaknesses
- The methodological contribution beyond the pretrained backbone is limited. DUET inherits its joint audio-video generator and flow-matching objective from Ovi, with the main additions being Whisper cross-attention, SeedVC-based timbre perturbation, and masked cross-modal interaction. These are reasonable adaptations, but the method is largely a new conditioning and training setup for an existing generator. My concern is not architectural simplicity, but that the paper does not demonstrate a clear technical advance beyond adapting the backbone to this task.
- The evaluation does not demonstrate that vocal identity is inferred from the image, as confounding factors remain. Further testing with independent recordings and models without visual conditioning is needed to attribute speaker similarity gains to visual identity inference.
- Perceptual compatibility is not adequately distinguished from individual vocal identity. The paper explicitly aims to infer the depicted person’s vocal identity, whereas age and gender agreement only assess demographic compatibility. The user study evaluates whether the generated voice matches the visible speaker without documenting the use of the actual target voice as a reference, so it cannot independently establish person-specific identity recovery.
- The attention formulation omits the handling of silent audio tokens. Under Eqs. (4) to (7), an audio token outside all speaking intervals has no permitted video keys, producing an attention row containing only negative-infinity logits when the transposed mask is applied. Ordinary softmax is undefined for such a row. The implementation may handle fully masked queries safely, but this behavior should be specified.
Questions for authors
- How much of the SECS improvement remains without visual conditioning, and does the improvement persist with independently recorded driving speakers?
- A portrait-swap experiment with fixed audio and text conditions would help determine whether the output accurately reflects the intended individual’s identity.
- Can the authors report per-speaker voice similarity, assignment accuracy, and unintended mouth motion during the other person’s speaking turn? These measurements would address the claimed binding capability more directly than the current metrics.