acceptodds
Under review as a conference paper at ICLR 2027

What Enables Motion Coordination in Talking Portrait Generation? From Cross-Domain Motion Representation to Alignment and Fusion

Abstract

3DGS-based talking portrait generation has achieved significant progress. However, motion coordination still presents challenges, particularly in the heterogeneous modality differences between audio and portraits, as well as in interactions among motion regions. This primarily stems from the fact that 3DGS is fundamentally based on multi-view appearance fitting, where portrait motion is coupled with appearance only implicitly or indirectly, leading to weakened and limited motion representations. To address these issues, we propose a cross-domain motion representation framework for talking portrait generation, aiming to move beyond the traditional paradigm that relies on appearance variations. Specifically, we construct motion representations in the audio-visual domain and the density domain by leveraging motion variation characteristics across modalities and portrait regions, thereby enhancing motion coordination. The framework consists of two key strategies: from audio-visual domain representation to alignment (AR-A) and from density domain representation to fusion (DR-F). For AR-A, we construct the audio-visual representation based on the correlation between audio and different portrait structures, and leverage strongly correlated structures to guide the alignment of weakly correlated ones, thereby enhancing cross-modal synchronization. For DR-F, we construct the density representation based on motion variations in continuous sequences, and map the motion sequences aligned in AR-A into density space for fusion, thereby enabling cross-region motion interaction. Experimental results demonstrate that our method achieves superior motion coordination and outperforms existing state-of-the-art approaches.

open until 14 Dec 2026

est. 32% chance this paper gets accepted at ICLR 2027.

Reject 68%Accept 32%

What do you think this paper will get?

All positions stay anonymous.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.