acceptodds
Withdrawn from ICLR 2027

Why Not Rethink Talking Portrait Generation through Dynamic Evolution? Motion Interaction Representation Makes It Possible

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

In recent years, 3DGS-based methods have made significant progress in portrait reconstruction and generation. However, existing methods still struggle to capture the interactions among multiple motion factors over time, often resulting in incoherent fine-grained dynamics and insufficient coordination across different motion components. The fundamental reason is that 3DGS adopts an explicit parameterized representation based on discrete Gaussian primitives, which primarily describes spatial states but lacks an effective mechanism for characterizing the interactions among motion factors and their state evolution. We argue that the key to dynamic portrait generation lies in learning an interaction-oriented state evolution mechanism that enables the model to predict future changes based on the current dynamic state, rather than relying on local observations for frame-by-frame fitting. To this end, we propose a talking portrait generation framework based on motion interaction representation, which establishes relationships among dynamic factors from two aspects: static-dynamic factor fusion (S-DF) and rigid-flexible factor alignment (R-FA). S-DF establishes the static-dynamic interaction representation to enhance the expression of fine-grained dynamic evolution, while R-FA develops the rigid-flexible interaction representation to enable coordinated dynamic changes across different regions. Extensive experiments on talking portrait generation demonstrate that our method effectively captures the interactions among motion factors, resulting in portrait videos with higher fidelity in fine-grained details and more natural motion coordination.

Primary area:
generative models
Keywords:
3DGS, Video Generation, Interaction, Fusion, Alignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bCaSOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a 3DGS-based talking portrait generation framework that models interactions among different motion factors. It introduces static-dynamic factor fusion (S-DF) to combine structural information with dynamic details, and rigid-flexible factor alignment (R-FA) to coordinate motion across different regions. These representations condition an MLP that updates Gaussian states over time. However, the primary focus of the paper is on how to decouple dynamic and static representations.

Critical strengths

Modeling the relationships between global motion and local facial details is a reasonable idea.

The qualitative examples and video results show promising tooth and wrinkle details, supported by component ablations.

Critical weaknesses

I cannot understand the audio-driven inference pipeline, even after reading the appendix. Both modules extract features from video frames, but how are these features obtained when driving the model with new audio? The audio encoder, conditioning mechanism, and initial Gaussian state are not clearly described. Please specify exactly what inputs are required during inference.

The experimental setup lacks essential details, including the number of identities, video durations, training/test splits, and whether each person requires separate training. It is unclear whether the reported results evaluate video reconstruction or generation from unseen audio. This makes the substantial improvements difficult to assess.

The losses do not sufficiently justify the claimed factor decomposition. Feature orthogonality does not necessarily separate static structure from motion, and increasing feature distances does not establish a rigid-to-flexible hierarchy. Also, the temporal loss encourages features at different times to be similar. What prevents it from suppressing meaningful motion?

The claim of predicting future states is unclear. The temporal Transformer processes a sequence without a specified causal mask. Does the representation used to predict the next state already contain information from future video frames?

Questions for authors

I find the central question very interesting: how can we disentangle static and dynamic features? This is a fundamental problem that extends well beyond talking head generation. While the proposed constraints seem intuitive, I have two main concerns:

There is insufficient analysis showing that disentanglement actually occurs. Better generation quality alone does not establish successful disentanglement. Could the authors provide feature visualizations, controlled feature-swapping experiments, or downstream evaluations to support this claim?

I have a more fundamental concern about the orthogonality objective. Two encoders could preserve the same information but use different final linear projections to place it in mutually orthogonal subspaces. The orthogonality loss could then be zero without any meaningful separation of static and dynamic information. What prevents this solution in the proposed framework?

I also find the temporal consistency loss insufficiently justified. Why should the features repeat over the chosen interval, and how is that interval selected? What makes this objective learn meaningful temporal structure rather than simply suppress changes? Are there prior studies supporting this design, or visualizations and downstream experiments demonstrating what it learns?

Finally, for an ICLR submission, the neural architecture and computational pipeline need to be clearly specified. I found extensive descriptions of the losses, but insufficient detail about the models implementing them. Please explain the network architectures, how audio enters the model, and the complete training and inference procedures.

Reviewer_BdKZOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses insufficient modeling of motion-factor interactions in 3DGS-based talking portrait generation with a framework centered on "motion interaction representation". Two complementary strategies—static-dynamic factor fusion (S-DF) and rigid-flexible factor alignment (R-FA)—jointly drive the recurrent evolution of Gaussian states. The authors compare with six 3DGS baselines on self-collected YouTube videos, with per-module ablations and a user study; the main gaps are the missing comparison with commercial/open-source systems, the unspecified audio pathway, and the incomplete evaluation protocol.

Critical strengths
  • Clear motivation and problem decomposition: the paper accurately notes that existing 3DGS methods lack mechanisms for motion-factor interactions and state evolution. Organizing the solution along the static-dynamic and rigid-flexible dimensions is convincing.

  • Symmetric framework with complete derivations: the four S-DF and four R-FA modules are each paired with loss constraints, and Appendix B gives step-by-step derivations. The derivation depth exceeds the main-text average, showing solid theoretical grounding.

  • Comprehensive ablations: Table 2 removes each of all 8 modules, with PSNR consistently dropping from 36.48 to 32.34–32.75. Every module's necessity is supported by both quantitative and qualitative evidence.

  • Leading results with high efficiency: the method leads on all reported metrics over the runner-up (e.g., PSNR 36.48) while reaching 163.7 FPS. This high-quality-plus-high-efficiency combination has clear practical value.

  • Good-faith appendix analyses: C.1 presents a 100-participant user study and C.3 systematically compares key parameter configurations. C.4 honestly shows failure cases (Figure 11), and the discussion is concrete rather than generic.

Critical weaknesses
  • No discussion or comparison with commercial-grade or open-source systems: the comparison is limited to six 3DGS academic baselines, while directly relevant systems such as MagicInfinite, MegActor-Sigma, and Livatar-1 are neither discussed nor compared. The "outperforming state-of-the-art" claim therefore holds only within the self-selected baseline subset, leaving the method's standing against the strongest practical systems unknown. Related references:

    • [1] Yi, H., Ye, T., Shao, S., Yang, X., Zhao, J., Guo, H., ... & Zhou, D. (2025). Magicinfinite: Generating infinite talking videos with your words and voice. arXiv preprint arXiv:2503.05978.
    • [2] Yang, S., Li, H., Wu, J., Jing, M., Li, L., Ji, R., ... & Wang, J. (2025, April). Megactor-sigma: Unlocking flexible mixed-modal control in portrait animation with diffusion transformer. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 39, No. 9, pp. 9256-9264).
    • [3] Liu, H., Hong, X., Yang, X., Ruan, Y., Lian, X., Lingelbach, M., ... & Li, W. (2025). Livatar-1: Real-Time Talking Heads Generation with Tailored Flow Matching. arXiv preprint arXiv:2507.18649.
  • The audio-driven pathway is entirely missing: the task and metrics target audio-lip synchronization, yet the method section contains no audio feature extraction or audio-visual fusion pathway. The origin of synchronization scores such as SyncNet = 7.942 is unexplained and must be clarified in the rebuttal.

  • Incomplete evaluation protocol and dataset information: training uses only self-collected YouTube videos, with no identity count, duration, or train/test split reported, and whether results are reconstruction or unseen-audio generation is unclear. Metrics claimed in the text (SSIM, AUE, FID) are absent from the result tables, raising fairness concerns.

Questions for authors
  • Would you add a discussion and effect comparison with systems such as MagicInfinite, MegActor-Sigma, and Livatar-1? On which shared metrics and evaluation protocols would comparison be fair?

  • How exactly does the audio signal enter the framework, and where is it fused with Z_S-DF/Z_R-FA or the state update? Are SyncNet/LSE-C computed on reconstructed training frames or on frames generated under unseen audio?

  • What are the values of the key hyperparameters (N, M, K, ∆, h, and the auxiliary loss weights)? How are the Gaussian primitives initialized, and does the recurrent update exhibit accumulated drift on sequences beyond one minute?

Reviewer_o7wEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a framework for talking-head generation based on motion-interaction representations. It aims to address the limitations of existing 3DGS methods, which struggle to effectively model the interactions between different motion factors and often lack consistency in fine-grained dynamic details.

The framework consists of two modules: (1) S-DF: it decouples static structures from dynamic representations, performs spatio-temporal decomposition, and integrates the resulting features. (2) R-FA: it coordinates motion variations across different regions through regional dynamic feature extraction, hierarchical decomposition, and adaptive weight prediction.

Experimental results demonstrate that this approach outperforms existing 3DGS-based talking-head generation methods. The authors further validate the effectiveness of each component through ablation studies and user evaluations.

Critical strengths
  • The motivation is convincing. Fine-grained facial dynamics depend on not only the independent movements of specific regions, but also the coordinated relationships between these movements, the subject's identity structure, their overall pose, and other facial areas. By modeling dynamic portrait generation through the lens of the interactions between various movement factors, this work have clear research value.

  • The paper compares the proposed method with some baselines, including GaussianTalker, InsTaG, MGGTalk, TaoAvatar, EmoTaG, and EmoDiffTalk. The proposed method achieves a state-of-the-art performance. Furthermore, the authors conduct ablation studies on various sub-modules of S-DF and R-FA, which helps in analyzing the contribution of each component to the final performance.

Critical weaknesses
  • The methodology section lacks a clear and coherent technical description. The authors introduce many modules for feature extraction, decomposition, fusion, and alignment, with many equations, but fail to provide a clear explanation of the relationships between these modules, the process of representation, or the specific motivations behind each design choice. For example, S-DF extracts static and dynamic representations directly from the input video sequence, while R-FA relies on input video frames to extract regional motion features, without any detail of these regions or their meaning. Although the overall framework diagram includes an audio input, method does not explicitly describe how audio features contribute to the construction of motion-interaction representations, nor how they drive the subsequent Gaussian state updates.

  • The design of the dynamic evolution mechanism, as well as its effectiveness, still requires further provement. Existing methods have not yet fully explained how S-DF ensure that the model learns valid patterns of dynamic evolution. For example, the temporal dynamic decomposition module employs a Transformer to extract distinct patterns of temporal variation, while enforcing consistency among dynamic features separated by fixed time intervals. The authors should provide more intuitive evidence to demonstrate that this decomposition effectively extracts these distinct patterns of variation. Similarly, Equation 14 encourages consistency in the motion variations of regions across different temporal positions. The authors need to demonstrate that these regions indeed extract change features at different levels, and clarify the underlying significance of these features.

  • R-FA decomposition lacks explicit constraints regarding motion properties. Existing descriptions are insufficient to demonstrate that the learned hierarchical representations truly correspond to rigid and flexible motions. While the authors employ a hierarchical decomposition module to map regional attention features into distinct motion levels based loss to constrain the distances between adjacent levels distance constraint does not inherently guarantee that these levels correspond to the physical properties ranging from rigid to flexible motion.

  • Experiments fail to sufficiently distinguish the performance gains attributable to motion interaction modeling itself from those resulting from general feature fusion or temporal modeling. Although Table 2 presents ablation studies for several sub-modules, these experiments primarily demonstrate that the respective components contribute to the final generation performance; they are insufficient to definitively prove that the motion interaction modeling mechanism is the primary driver of the performance improvements. It is recommended to include more targeted comparisons, by removing the complete S-DF and R-FA branches, or by benchmarking against standard temporal feature fusion and conditional deformation prediction methods with a similar number of parameters to further validate the necessity and unique contribution of the interaction mechanism itself.

Questions for authors

No additional questions, the rebuttal is expected to address the concerns above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.