acceptodds
Under review as a conference paper at ICLR 2027

Streaming End2End Audio-Visual Identity Swapping for Talking Videos

Abstract

Talking-video character replacement requires coordinated transfer of appearance and voice while preserving the source motion, scene, linguistic content, and audio-video timing. Existing methods use separately optimized models for the two modalities, making audio-visual consistency difficult to enforce. We present UniSwap, the first framework for streaming joint audio-visual identity replacement in talking videos. Given a source video, a reference image, and a reference voice clip, UniSwap transfers the reference appearance and vocal timbre within a single audio-visual diffusion transformer while preserving the source content and dynamics. To address the scarcity of aligned cross-identity training pairs, we introduce a swap-and-reconstruct pipeline that synthesizes identity-mismatched yet motion-preserving source videos with learned replacement models and reconstructs the original clips under reference guidance, exposing the model exclusively to natural video conditioning without estimated pose, landmark, or segmentation inputs. Starting from a bidirectional backbone, we progressively adapt the model through In-context Pretraining for joint replacement, Conditional Streaming Adaptation for block-causal KV-cached generation, and Efficient Self-forcing DMD for mitigating exposure bias while reducing sampling from 30 to 3 denoising steps per block; Efficient Multi-LoRA Switching lets the three DMD roles share a single frozen backbone. Feature-RoPE Decomposition keeps cached positions within the training range, supporting stable long-form inference. Experiments demonstrate strong audio-visual synchronization, competitive identity preservation, efficient streaming, and stable long-form generation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Character Replacement, Joint Audio-Visual Generation, Streaming Video generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_aphXOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2(−1)
Soundness
2
Read review
Paper summary

This paper introduces UniSwap, a framework for jointly replacing appearance and voice in talking videos. Given a source video, a reference image, and a reference voice clip, the model transfers the target identity while preserving the source content and motion. The authors construct training pairs through a swap-and-reconstruct pipeline. They then adapt an audio-video diffusion model into a streaming generator through three training stages, with cache and positional encoding designs to support longer videos.

Critical strengths
  1. Jointly replacing appearance and voice is a useful task. Integrating both into one model is an interesting alternative to combining separate video replacement and voice conversion systems.

  2. I think the swap-and-reconstruct pipeline is intuitive. It prov2. ides paired supervision while keeping real videos as the reconstruction targets.

  3. The authors provide ablations for the cache and positional encoding designs, which help explain their contributions to stable generation.

Critical weaknesses
  1. My primary concern is poor lip synchronization in demos. There is clear misalignment between speech and mouth movements, contradicting the reported metrics. Since audio-video consistency is a core goal, this undermines confidence in the method. The authors should explain the gap between quantitative results and demos, and evaluate these failure cases. Frequent hand artifacts are also observed, with unnatural intertwined or merged fingers. These artifacts distract viewers, despite the method’s goal to preserve source motion.

  2. The real-time and long-video claims lack sufficient evidence. The reported 23.5 FPS is throughput on three H100 GPUs, while actual input-output latency is unknown. The first-frame generation time and streaming delay should be provided. Figure 1 claims hour-scale generation, but long-video evaluation only uses 1-minute samples. Longer video results are required.

Questions for authors

See Critical Weaknesses.

Reviewer_8TC8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces UniSwap, a unified framework for streaming audio-visual identity replacement in talking videos. Given a source video, a reference image, and a reference voice clip, the method jointly transfers visual appearance and vocal identity while preserving source motion, background, and linguistic content.

The approach introduces a swap-and-reconstruct strategy for generating aligned training pairs, followed by three-stage adaptation of a diffusion transformer. It combines in-context pretraining, conditional streaming adaptation, and self-forcing distillation to enable efficient autoregressive generation. Feature-RoPE Decomposition is introduced to improve temporal consistency during long-form inference.

Experiments demonstrate improved audio-visual synchronization, competitive identity preservation, and streaming throughput of 23.5 FPS using three H100 GPUs. However, several visual and voice quality metrics remain below specialized baselines.

Critical strengths
  1. Unified Audio-Visual Identity Replacement

The paper addresses an important limitation of existing approaches by jointly replacing visual and vocal identities within a single generative framework. Unlike independently optimized pipelines, UniSwap enables coordinated generation across modalities. The improved synchronization metrics provide empirical support for this unified formulation.

  1. Efficient Streaming Generation and Long-Form Consistency

The integration of block-causal generation, self-forcing distillation, and Feature-RoPE Decomposition enables efficient streaming inference. The reported throughput of 23.5 FPS is promising, and the ablation experiments demonstrate the effectiveness of the proposed memory and positional encoding mechanisms in maintaining temporal consistency.

  1. Practical Training Data Construction

The swap-and-reconstruct strategy provides a practical solution to the scarcity of aligned cross-identity audio-visual training pairs. By generating synthetic identity-swapped sources and reconstructing the original videos, the method avoids requiring different individuals to perform identical motions and speech. The construction of approximately 100K filtered training samples demonstrates the scalability and practical value of this approach.

Critical weaknesses
  1. Insufficient Validation of Long-Form Streaming

Although the paper emphasizes long-form generation and suggests support for hour-scale videos, quantitative evaluation is restricted to 30 one-minute sequences. This is insufficient to establish reliability over longer durations, particularly given potential autoregressive error accumulation. Furthermore, the reported 23.5 FPS throughput requires three H100 GPUs, while startup latency and end-to-end streaming delay are not reported. These limitations weaken the practical streaming claims.

  1. Limited Performance Advantages Over Existing Baselines

While UniSwap achieves stronger synchronization, it underperforms specialized baselines on several important metrics. For example, its DINO-S score (0.596) is below SCAIL-2 (0.630), and its speaker similarity (0.808) is below Seed-VC (0.829). The user study also reports lower voice identity ratings than the evaluated cascaded approaches. Moreover, the comparison does not include cascaded pipelines equipped with explicit lip synchronization refinement, making it difficult to determine whether unified generation provides advantages beyond synchronization.

  1. Limited Evidence of Generalization and Contribution Isolation

The training pipeline relies heavily on synthetic identity-swapped videos generated using existing replacement and voice conversion models. Although quality filtering is applied, the paper does not explicitly quantify the domain gap between synthetic training conditions and real cross-identity inference. Additionally, several components build on existing diffusion distillation and attention mechanisms, and the available ablations do not fully isolate their individual contributions against alternative designs. This makes it difficult to assess the generality and methodological novelty of the complete framework.

Questions for authors
  1. Can the authors provide quantitative evaluations on substantially longer videos, such as 5, 15, or 30 minutes? This would help establish whether the proposed framework prevents identity drift and accumulated generation errors beyond the currently evaluated one-minute sequences.

  2. How does UniSwap compare with cascaded approaches that incorporate explicit lip synchronization refinement after voice conversion? Such experiments would help isolate the benefits of unified audio-visual generation from those of synchronization correction.

  3. How sensitive is the proposed framework to the quality and distribution of the synthetically generated training sources? Can the authors provide additional experiments evaluating generalization to challenging real-world inputs, including substantial identity differences, diverse recording conditions, and imperfect reference samples?

  4. Can the authors report startup latency, end-to-end delay, and a detailed comparison of generation quality across the three training stages? Table 4 indicates that the final streaming model underperforms Stage 1 on several quality metrics, and understanding this trade-off would help assess the practical benefits of the proposed adaptation strategy.

Reviewer_6u5kOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

UniSwap jointly replaces appearance and voice in talking videos while retaining source motion, scene, and speech content. It constructs training pairs through swap-and-reconstruct synthesis and adapts an audio-video diffusion backbone through in-context training, causal streaming adaptation, and self-forcing distillation. A bounded KV cache with positional re-anchoring enables long-form generation. Experiments evaluate synchronization, identity, perceptual quality, speech intelligibility, and streaming throughput.

Critical strengths

Joint appearance and voice replacement is a useful task, and the results show a synchronization benefit over the evaluated cascades. On 100 held-out clips, UniSwap achieves the best Sync-C and Sync-D, while maintaining competitive visual and speaker identity similarity. The blinded user study also favors its lip synchronization, although it covers only four source clips.

The streaming pipeline is supported by meaningful ablations. Removing bounded positional encoding substantially degrades synchronization and voice similarity over one-minute sequences, while removing the sink block causes visual identity drift. The three-step generator and detailed runtime breakdown make the system contribution concrete.

Critical weaknesses

The real-time and duration claims exceed the reported evidence. The 23.5 FPS result measures steady-state throughput on three H100 GPUs, not input-to-output latency, and falls below the 25 FPS used to standardize the data. Source-block buffering and pipeline delay are not reported as an end-to-end latency measurement. Likewise, Figure 1 advertises hour-scale generation, while the long-video benchmark contains only one-minute clips. These distinctions matter for the claimed interactive and long-form applications.

The contribution of joint modeling is not isolated from other system differences. Comparisons use external visual models paired with Seed-VC, but there is no matched-backbone ablation of cross-modal interaction. UniSwap improves synchronization while trailing stronger baselines on visual quality and DNSMOS, and the user study rates its voice identity lowest among the compared systems. A controlled ablation would help establish which gains come from joint optimization and how they trade off against identity and quality.

Relevant prior work is missing. DreamID-Omni addresses joint visual identity and voice control, while UnityShots, ShotStream, and Vorch-Director study related memory, streaming, and long-form consistency mechanisms. The paper should discuss these works to clarify UniSwap’s distinct contributions.

Questions for authors

What are the measured startup and steady-state input-to-output latencies, including source buffering, encoding, generation, and decoding? Can the system sustain a 25 FPS input without accumulating delay?

Is there quantitative evidence for hour-scale stability beyond the one-minute benchmark? If not, can the duration claim be restricted to the evaluated setting?

Can you provide a matched-backbone comparison with cross-modal interaction disabled, reporting synchronization, identity similarity, and perceptual quality?

How does UniSwap differ from DreamID-Omni in its supported identity-editing setting and cross-modal conditioning? Could DreamID-Omni be evaluated under compatible inputs, or could you clarify what prevents a fair comparison?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.