acceptodds
Withdrawn from ICLR 2027

What Enables Motion Coordination in Talking Portrait Generation? From Cross-Domain Motion Representation to Alignment and Fusion

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

3DGS-based talking portrait generation has achieved significant progress. However, motion coordination still presents challenges, particularly in the heterogeneous modality differences between audio and portraits, as well as in interactions among motion regions. This primarily stems from the fact that 3DGS is fundamentally based on multi-view appearance fitting, where portrait motion is coupled with appearance only implicitly or indirectly, leading to weakened and limited motion representations. To address these issues, we propose a cross-domain motion representation framework for talking portrait generation, aiming to move beyond the traditional paradigm that relies on appearance variations. Specifically, we construct motion representations in the audio-visual domain and the density domain by leveraging motion variation characteristics across modalities and portrait regions, thereby enhancing motion coordination. The framework consists of two key strategies: from audio-visual domain representation to alignment (AR-A) and from density domain representation to fusion (DR-F). For AR-A, we construct the audio-visual representation based on the correlation between audio and different portrait structures, and leverage strongly correlated structures to guide the alignment of weakly correlated ones, thereby enhancing cross-modal synchronization. For DR-F, we construct the density representation based on motion variations in continuous sequences, and map the motion sequences aligned in AR-A into density space for fusion, thereby enabling cross-region motion interaction. Experimental results demonstrate that our method achieves superior motion coordination and outperforms existing state-of-the-art approaches.

Primary area:
generative models
Keywords:
3DGS, Video Generation, Multimodality, Alignment, Fusion

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8iCjOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies audio-driven talking portrait generation using 3D Gaussian Splatting (3DGS), focusing on audio–lip synchronization and coherent motion across facial regions. The proposed framework consists of two components: audio-visual representation and alignment (AR-A), which aims to exploit differences in audio correlation across portrait structures, and density-domain representation and fusion (DR-F), which organizes motion features hierarchically to facilitate coordinated motion. The resulting representations condition an MLP that predicts updates to Gaussian parameters for rendering.

Critical strengths

Relevant problem formulation. Beyond accurate lip synchronization, coordinating movements across the mouth, eyes, eyebrows, and head is an important objective for natural talking portrait generation. Potential practical value. The reported rendering performance suggests potential for applications requiring efficient animation of personalized avatars, provided that the end-to-end inference cost and identity-specific preparation requirements are clarified. Complementary modeling objectives. Separately addressing audio-visual alignment and motion coordination is conceptually reasonable. The paper includes quantitative comparisons and component ablations intended to evaluate these objectives.

Critical weaknesses

The motivation for the chosen framework is insufficiently established. The paper largely takes a 3DGS-based formulation as its starting point without adequately explaining its intended advantages relative to diffusion-based talking portrait and video generation approaches. It remains unclear whether the primary benefit is lower latency, explicit 3D control, identity consistency, or adaptation efficiency. The reported FPS alone does not establish this advantage without clarification of preprocessing, identity-specific training, inference inputs, and end-to-end runtime. A comparison under compatible task settings, or a more precise discussion of the relevant trade-offs, is needed.

The methodological novelty is not sufficiently demonstrated. The framework primarily combines convolutional networks, MLPs, cross-modal attention, and feature alignment and consistency losses. The use of established components is not itself a limitation; however, the paper does not clearly identify the substantive technical advance beyond their combination. In particular, the mechanism by which strongly audio-correlated structures guide weakly correlated structures remains underspecified. Likewise, the “density” representation is not sufficiently distinguished from a conventional hierarchical latent representation. Comparisons against simpler, capacity-matched alignment and temporal fusion alternatives would help establish the necessity of the proposed design.

The empirical evidence does not clearly validate the central contribution. In Table 2, removing the alignment module increases SyncNet from 7.625 to 8.507 and LSE-C from 7.562 to 8.094, although both metrics are marked as higher-is-better. This contradicts the claim that the module improves synchronization. Furthermore, improvements in reconstruction quality do not by themselves demonstrate improved cross-region motion coordination. These issues weaken the evidence that the proposed components provide the claimed benefits.

Questions for authors
  1. What is the intended application setting, and what specific advantage does the proposed framework offer over diffusion-based alternatives under comparable input and personalization requirements?
  2. What inputs are required at inference time? In particular, how are the visual features and temporal frame windows obtained when generating a video from previously unseen audio?
  3. What precisely constitutes the “density” representation, and how does it differ from standard hierarchical motion features? Where is the strong-to-weak structural guidance explicitly implemented?
  4. Can the authors explain the apparent contradiction between the synchronization results in Table 2 and the accompanying claims?
  5. Can the authors provide comparisons with simpler, capacity-matched alternatives and direct evaluations of cross-region motion coordination?
Reviewer_9Mz9OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a cross-domain motion representation framework for 3D Gaussian Splatting (3DGS)-based talking portrait generation to improve motion coordination across modalities and portrait regions. The framework consists of two components: Audio-Visual Representation to Alignment (AR-A), which leverages correlations between audio and different portrait structures to guide cross-modal alignment, and Density Representation to Fusion (DR-F), which maps temporally aligned motion sequences into the density domain to facilitate interactions across different motion regions. By explicitly modeling motion representations beyond appearance variations, the proposed method aims to improve cross-modal synchronization and cross-region motion coordination. Experimental results demonstrate that the method outperforms existing state-of-the-art approaches in motion coordination.

Critical strengths
  1. Propose a framework that conduct cross-domain motion representation for 3DGS-based talking portrait generation.
  2. Propose Audio-visual representation for cross-modal alignment and density representation to fuse different motion regions.
Critical weaknesses
  1. Insufficient analysis of cross-modal interactions. In Lines 169–175, the authors state that the learned attention mechanism can “capture the cross-modal interactions between audio and visual features.” However, it is unclear how this claim is validated. The authors should provide visualization and analysis of the learned attention weights to demonstrate whether the model captures meaningful cross-modal relationships and whether these relationships vary across different identities.
  2. Unclear role of F_{\text{denoise}}. The purpose of F_{\text{denoise}} in Lines 186–187 is unclear. If this feature can be obtained through fitting during data preparation, why is an additional denoising module necessary? Furthermore, the rationale for using U to represent G is not clear and should be explained.
  3. Inconsistencies in the reported results and analysis. In Table 2, the proposed method does not achieve the best results on either Sync or LSE-C. However, the authors claim that removing F_{\text{multimodal}} leads to degraded lip synchronization, but Table 2 shows that both Sync and LSE-C improve when this component is removed. The authors should clarify this discrepancy and provide a more accurate interpretation of the results.
  4. Several important implementation details are missing. For a talking-head generation method, it is unclear how audio drives the generated facial motion. The paper also lacks experiments on audio-driven animation across different identities, making it difficult to assess the generalization ability of the method. Furthermore, the authors do not clarify whether the model is trained for a single identity or jointly trained on multiple identities. The treatment of the background during training and inference is also not described.
  5. The paper reports substantially higher PSNR than the baselines, but does not provide sufficient details about the evaluation protocol, including the number of training and evaluation identities, whether test identities are held out, the video duration and amount of training data available per identity, or whether the few-shot baselines (InsTaG and EmoTaG) are retrained using the same data budget. Without this information, potential identity leakage and inconsistencies in the evaluation protocols cannot be ruled out. The authors should disclose the identity splits and the data budget used for each method to ensure a fair comparison. In addition, Fig. 4 shows noticeable differences between the generated results and the ground truth, which raises questions about how the proposed method achieves such high PSNR.
Questions for authors

The motivation behind Eq. (8) is unclear. Audio and visual features have complex many-to-many correspondences, so it is unclear whether the proposed direct training strategy is appropriate for modeling these relationships. The authors should better justify this design and explain how it handles the ambiguity and diversity of audio-visual correspondences.

Reviewer_nnStOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a 3D Gaussian talking-portrait framework with audio-visual representation and alignment (AR-A), and hierarchical motion-density representation and fusion (DR-F). These features condition Gaussian updates to coordinate facial and regional motion. The paper reports comparisons with six methods, component ablations, qualitative examples, and a user study. Assessing the claimed audio-driven improvement requires clarification of the synchronization results, test-time inputs, and the relationship between the supplied prototype and the evaluated system.

Critical strengths
  1. A relevant coordination problem. Distinguishing strongly audio-correlated articulation from less directly constrained regional motion is useful. The proposed decomposition attempts to address coordination beyond lip appearance, a practically important goal for personalized avatars.

  2. Attention to observable quality and practical operation. The qualitative examples focus on wrinkles, hair, occlusion, and challenging poses, and the evaluation attempts to cover visual fidelity, synchronization, identity, and speed. These provide useful targets for assessing the method. The reported consumer-GPU setup is potentially accessible, although the implementation and protocol questions below prevent treating this as demonstrated reproducibility.

Critical weaknesses
  1. The synchronization ablation contradicts the stated interpretation. In Table 2, both SyncNet and LSE-C are marked higher-is-better. Removing alignment increases them from 7.625/7.562 to 8.507/8.094, and other AR-A removals show similar increases. The text instead says these removals impair synchronization. This may be a reporting error or a quality–synchronization trade-off, but currently the table does not establish the claimed synchronization benefit. The image-quality improvements should be acknowledged separately.

  2. The evaluated task and implementation are not sufficiently connected. Training/test identity counts, clip durations, splits, and the visual inputs available during novel-audio inference are unclear. The supplied code takes video frames, generates dummy audio, leaves rendering unimplemented, and omits rendered-image supervision from its implemented total loss, despite Appendix B.10 including it. These are larger gaps than small configuration bugs. A schematic prototype cannot verify the reported portrait quality, audio-driven behavior, or runtime; this does not establish that the actual experiments were invalid.

Questions for authors
  1. Results correction: Which Table 2 values, metric directions, or interpretations require correction? If the complete model trades synchronization for fidelity, please quantify and explain that trade-off rather than claiming uniform improvement.

  2. Task protocol: Please list every test-time visual input and its provenance. Are target-time RGB frames required, or can the system generate from a fixed identity representation and new audio? Please provide dataset sizes, held-out splits, and matched identity-specific training budgets for the baselines.

  3. Implementation correspondence: Is the supplementary code intentionally schematic? Please identify or provide the minimal implementation corresponding to the reported results, including real audio processing, point-conditioned Gaussian updates, rendering, reconstruction supervision, and evaluation settings. A corrected tensor shape alone would not resolve these gaps.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.