- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2(−1)
- Soundness
- 2
Read review
Paper summary
This paper introduces UniSwap, a framework for jointly replacing appearance and voice in talking videos. Given a source video, a reference image, and a reference voice clip, the model transfers the target identity while preserving the source content and motion. The authors construct training pairs through a swap-and-reconstruct pipeline. They then adapt an audio-video diffusion model into a streaming generator through three training stages, with cache and positional encoding designs to support longer videos.
Critical strengths
-
Jointly replacing appearance and voice is a useful task. Integrating both into one model is an interesting alternative to combining separate video replacement and voice conversion systems.
-
I think the swap-and-reconstruct pipeline is intuitive. It prov2. ides paired supervision while keeping real videos as the reconstruction targets.
-
The authors provide ablations for the cache and positional encoding designs, which help explain their contributions to stable generation.
Critical weaknesses
-
My primary concern is poor lip synchronization in demos. There is clear misalignment between speech and mouth movements, contradicting the reported metrics. Since audio-video consistency is a core goal, this undermines confidence in the method. The authors should explain the gap between quantitative results and demos, and evaluate these failure cases. Frequent hand artifacts are also observed, with unnatural intertwined or merged fingers. These artifacts distract viewers, despite the method’s goal to preserve source motion.
-
The real-time and long-video claims lack sufficient evidence. The reported 23.5 FPS is throughput on three H100 GPUs, while actual input-output latency is unknown. The first-frame generation time and streaming delay should be provided. Figure 1 claims hour-scale generation, but long-video evaluation only uses 1-minute samples. Longer video results are required.
Questions for authors
See Critical Weaknesses.