- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a 3DGS-based talking portrait generation framework that models interactions among different motion factors. It introduces static-dynamic factor fusion (S-DF) to combine structural information with dynamic details, and rigid-flexible factor alignment (R-FA) to coordinate motion across different regions. These representations condition an MLP that updates Gaussian states over time. However, the primary focus of the paper is on how to decouple dynamic and static representations.
Critical strengths
Modeling the relationships between global motion and local facial details is a reasonable idea.
The qualitative examples and video results show promising tooth and wrinkle details, supported by component ablations.
Critical weaknesses
I cannot understand the audio-driven inference pipeline, even after reading the appendix. Both modules extract features from video frames, but how are these features obtained when driving the model with new audio? The audio encoder, conditioning mechanism, and initial Gaussian state are not clearly described. Please specify exactly what inputs are required during inference.
The experimental setup lacks essential details, including the number of identities, video durations, training/test splits, and whether each person requires separate training. It is unclear whether the reported results evaluate video reconstruction or generation from unseen audio. This makes the substantial improvements difficult to assess.
The losses do not sufficiently justify the claimed factor decomposition. Feature orthogonality does not necessarily separate static structure from motion, and increasing feature distances does not establish a rigid-to-flexible hierarchy. Also, the temporal loss encourages features at different times to be similar. What prevents it from suppressing meaningful motion?
The claim of predicting future states is unclear. The temporal Transformer processes a sequence without a specified causal mask. Does the representation used to predict the next state already contain information from future video frames?
Questions for authors
I find the central question very interesting: how can we disentangle static and dynamic features? This is a fundamental problem that extends well beyond talking head generation. While the proposed constraints seem intuitive, I have two main concerns:
There is insufficient analysis showing that disentanglement actually occurs. Better generation quality alone does not establish successful disentanglement. Could the authors provide feature visualizations, controlled feature-swapping experiments, or downstream evaluations to support this claim?
I have a more fundamental concern about the orthogonality objective. Two encoders could preserve the same information but use different final linear projections to place it in mutually orthogonal subspaces. The orthogonality loss could then be zero without any meaningful separation of static and dynamic information. What prevents this solution in the proposed framework?
I also find the temporal consistency loss insufficiently justified. Why should the features repeat over the chosen interval, and how is that interval selected? What makes this objective learn meaningful temporal structure rather than simply suppress changes? Are there prior studies supporting this design, or visualizations and downstream experiments demonstrating what it learns?
Finally, for an ICLR submission, the neural architecture and computational pipeline need to be clearly specified. I found extensive descriptions of the losses, but insufficient detail about the models implementing them. Please explain the network architectures, how audio enters the model, and the complete training and inference procedures.