- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper presents AudioLift, which is a two-stage method for audio-driven full-head 3D Gaussian avatar generation from a single portrait without a parametric face model or person-specific optimization. The first stage employs a video diffusion model to produce five synchronized views (one frontal, four oblique) with self-attention across views. The second stage lifts these views into a 3D Gaussian representation at each time step, with improved robustness via view-aware corruption, active-volume regularization, and rear-view distillation.
This method is validated on RenderMe-360 and HDTF with comparisons to Real3D-Portrait, SDTalk, and NeRFFaceSpeech. The supplementary materials include video comparisons that examine temporal and multiview performance, including a couple long sequences.
Critical strengths
A critical strength of this paper is its clean and simple architecture. It turns the difficult problem of directly predicting 3D facial motion from audio into something more manageable by breaking it into two simpler subproblems, namely generating the facial performance in image space and then reconstructing 3D from those generated views. The multiple generated views are nicely synchronized through the existing self-attention within a video transformer. Also, the additional reconstruction components are well-motivated to address cross-view inconsistencies in the generated multiview inputs. Overall, I find this to be an elegant solution.
Another critical strength is the solid experimental results. AudioLift compares favorably against the strong baselines Real3D-Portrait, SDTalk, and NeRFFaceSpeech. Besides positive quantitative results, the visual comparisons in the supplementary material are also good. The temporal coherence and expressive facial motion across changing viewpoints gives solid support for AudioLift, despite some visible artifacts around the hair and neck.
Critical weaknesses
Although a good set of baseline methods is used for experimental comparisons, the omission of VASA-3D is significant in my view. VASA-3D is a recent work that also leverages a 2D audio-driven generative model for generating 3D audio-driven Gaussian head avatars from a single image. VASA-3D produces compelling results that I consider stronger than the three baselines used in this submission. MVP4D and CAP4D are also recent relevant works given their multiview generation for 4D avatar construction, though not as necessary for comparison since they rely on pose/expression or 3DMM controls. To me, the results are not quite fully convincing without a VASA-3D comparison.
Questions for authors
Why is VASA-3D not included as a baseline? As mentioned above, the experimental comparisons would be more convincing to me if a clear advantage over VASA-3D were demonstrated. If there is a reason that such a comparison is not feasible, it should be explained.
3 comments
I appreciate the authors' detailed rebuttal. I'm reading it carefully
Given the lack of publicly available code/models for VASA-3D, I agree that a controlled comparison should not be required. The clarifications on the differences between AudioLift and VASA-3D, specifically no per-identity optimization and no parametric face model, as well as reconstruction of the back of the head, help to better position this work. As VASA-3D is the most closely related work but hard to compare against empirically, I think a detailed discussion of these differences is essential to add to the paper.
Since a VASA-3D comparison is not included, I think the scope of empirical claims may need to be carefully recalibrated. The paper contains empirical claims relative to "existing methods", though these methods do not include VASA-3D.
Response to Reviewer 31Yg
We sincerely thank you for your thoughtful and encouraging review. We greatly appreciate your recognition of AudioLift as a clean and simple architecture and an elegant solution, and of the solid experimental results, including the temporal coherence and expressive facial motion across changing viewpoints in the supplementary videos. We also thank you for your constructive comment on the VASA-3D comparison, which we address below.
Response:
We agree that VASA-3D is the most closely related recent work. Unfortunately, VASA-3D is not open-sourced, so it cannot be added as a baseline: the official page states that its models and APIs are not publicly available, the VASA-1 model it depends on is likewise unreleased, and we are not aware of a third-party reimplementation. Reimplementing it ourselves is not possible either, because VASA-3D is trained on videos synthesized by VASA-1 and driven by VASA-1 motion latents, so any substitute would no longer be VASA-3D. A direct comparison with VASA-3D is therefore not feasible. MVP4D is not open-sourced either. The released CAP4D takes FLAME expression and pose parameters rather than audio as its driving input; since its inputs differ from AudioLift's, we did not include it in our audio-driven protocol. To the best of our knowledge, besides the three original baselines (Real3D-Portrait, SDTalk, NeRFFaceSpeech), the only additional released audio-driven 3D method we could find whose inputs and outputs match AudioLift's (one image plus audio in, a 3D head renderable from free viewpoints out) is LAM (He et al., 2025), the released LAM-20K model with its official Audio2Expression module. We additionally evaluated LAM using the same reference portraits, audio segments, and scoring protocol, and provide the results in the following table:
AudioLift outperforms LAM on PSNR, SSIM, and LPIPS on both datasets. Note that LAM fits FLAME to the reference image and thus remains a parametric method with per-reference fitting.
We also clarify the difference between AudioLift and VASA-3D. VASA-3D optimizes a person-specific Gaussian avatar bound to FLAME for every identity, then drives it with VASA-1 motion latents, and does not model the back of the head. AudioLift uses no parametric face model and no per-identity optimization: Stage I generates five time-aligned views directly from one portrait and the audio, and Stage II reconstructs the Gaussians of every frame in one feed-forward pass. Additionally, AudioLift reconstructs the full head including the rear view, as shown in Figure 4 of the paper. We will include this discussion in the revised version.