- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies audio-driven talking portrait generation using 3D Gaussian Splatting (3DGS), focusing on audio–lip synchronization and coherent motion across facial regions. The proposed framework consists of two components: audio-visual representation and alignment (AR-A), which aims to exploit differences in audio correlation across portrait structures, and density-domain representation and fusion (DR-F), which organizes motion features hierarchically to facilitate coordinated motion. The resulting representations condition an MLP that predicts updates to Gaussian parameters for rendering.
Critical strengths
Relevant problem formulation. Beyond accurate lip synchronization, coordinating movements across the mouth, eyes, eyebrows, and head is an important objective for natural talking portrait generation. Potential practical value. The reported rendering performance suggests potential for applications requiring efficient animation of personalized avatars, provided that the end-to-end inference cost and identity-specific preparation requirements are clarified. Complementary modeling objectives. Separately addressing audio-visual alignment and motion coordination is conceptually reasonable. The paper includes quantitative comparisons and component ablations intended to evaluate these objectives.
Critical weaknesses
The motivation for the chosen framework is insufficiently established. The paper largely takes a 3DGS-based formulation as its starting point without adequately explaining its intended advantages relative to diffusion-based talking portrait and video generation approaches. It remains unclear whether the primary benefit is lower latency, explicit 3D control, identity consistency, or adaptation efficiency. The reported FPS alone does not establish this advantage without clarification of preprocessing, identity-specific training, inference inputs, and end-to-end runtime. A comparison under compatible task settings, or a more precise discussion of the relevant trade-offs, is needed.
The methodological novelty is not sufficiently demonstrated. The framework primarily combines convolutional networks, MLPs, cross-modal attention, and feature alignment and consistency losses. The use of established components is not itself a limitation; however, the paper does not clearly identify the substantive technical advance beyond their combination. In particular, the mechanism by which strongly audio-correlated structures guide weakly correlated structures remains underspecified. Likewise, the “density” representation is not sufficiently distinguished from a conventional hierarchical latent representation. Comparisons against simpler, capacity-matched alignment and temporal fusion alternatives would help establish the necessity of the proposed design.
The empirical evidence does not clearly validate the central contribution. In Table 2, removing the alignment module increases SyncNet from 7.625 to 8.507 and LSE-C from 7.562 to 8.094, although both metrics are marked as higher-is-better. This contradicts the claim that the module improves synchronization. Furthermore, improvements in reconstruction quality do not by themselves demonstrate improved cross-region motion coordination. These issues weaken the evidence that the proposed components provide the claimed benefits.
Questions for authors
- What is the intended application setting, and what specific advantage does the proposed framework offer over diffusion-based alternatives under comparable input and personalization requirements?
- What inputs are required at inference time? In particular, how are the visual features and temporal frame windows obtained when generating a video from previously unseen audio?
- What precisely constitutes the “density” representation, and how does it differ from standard hierarchical motion features? Where is the strong-to-weak structural guidance explicitly implemented?
- Can the authors explain the apparent contradiction between the synchronization results in Table 2 and the accompanying claims?
- Can the authors provide comparisons with simpler, capacity-matched alternatives and direct evaluations of cross-region motion coordination?