acceptodds
Under review as a conference paper at ICLR 2027

AudioLift: Audio-Driven Single Image to Full-Head Gaussian Avatar via Multi-view Video Generation

Abstract

Existing 3D talking avatar methods mainly rely on parametric face models including 3DMM or FLAME, whose compact motion representations limit the models ability to synthesize extreme expressions and facial accessories modeling. We introduce AudioLift, a two-stage framework that uses generated multi-view videos as the intermediate representation for audio-driven avatar generation without parametric face models or per-instance optimization. Our framework combines a synchronized multi-view video generation stage with an inconsistency-robust Gaussian reconstruction stage to recover coherent full-head geometry from a single portrait image. Extensive experiments demonstrate that AudioLift produces 3D talking avatars capable of capturing more extreme facial expressions than existing methods, with improved geometric completeness and strong multi-view consistency across the full viewing range.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
audio-driven avatars, 3D Gaussian Splatting, talking head generation, multi-view video generation, novel view synthesis, video diffusion models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_31YgOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents AudioLift, which is a two-stage method for audio-driven full-head 3D Gaussian avatar generation from a single portrait without a parametric face model or person-specific optimization. The first stage employs a video diffusion model to produce five synchronized views (one frontal, four oblique) with self-attention across views. The second stage lifts these views into a 3D Gaussian representation at each time step, with improved robustness via view-aware corruption, active-volume regularization, and rear-view distillation.

This method is validated on RenderMe-360 and HDTF with comparisons to Real3D-Portrait, SDTalk, and NeRFFaceSpeech. The supplementary materials include video comparisons that examine temporal and multiview performance, including a couple long sequences.

Critical strengths

A critical strength of this paper is its clean and simple architecture. It turns the difficult problem of directly predicting 3D facial motion from audio into something more manageable by breaking it into two simpler subproblems, namely generating the facial performance in image space and then reconstructing 3D from those generated views. The multiple generated views are nicely synchronized through the existing self-attention within a video transformer. Also, the additional reconstruction components are well-motivated to address cross-view inconsistencies in the generated multiview inputs. Overall, I find this to be an elegant solution.

Another critical strength is the solid experimental results. AudioLift compares favorably against the strong baselines Real3D-Portrait, SDTalk, and NeRFFaceSpeech. Besides positive quantitative results, the visual comparisons in the supplementary material are also good. The temporal coherence and expressive facial motion across changing viewpoints gives solid support for AudioLift, despite some visible artifacts around the hair and neck.

Critical weaknesses

Although a good set of baseline methods is used for experimental comparisons, the omission of VASA-3D is significant in my view. VASA-3D is a recent work that also leverages a 2D audio-driven generative model for generating 3D audio-driven Gaussian head avatars from a single image. VASA-3D produces compelling results that I consider stronger than the three baselines used in this submission. MVP4D and CAP4D are also recent relevant works given their multiview generation for 4D avatar construction, though not as necessary for comparison since they rely on pose/expression or 3DMM controls. To me, the results are not quite fully convincing without a VASA-3D comparison.

Questions for authors

Why is VASA-3D not included as a baseline? As mentioned above, the experimental comparisons would be more convincing to me if a clear advantage over VASA-3D were demonstrated. If there is a reason that such a comparison is not feasible, it should be explained.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 31Yg

We sincerely thank you for your thoughtful and encouraging review. We greatly appreciate your recognition of AudioLift as a clean and simple architecture and an elegant solution, and of the solid experimental results, including the temporal coherence and expressive facial motion across changing viewpoints in the supplementary videos. We also thank you for your constructive comment on the VASA-3D comparison, which we address below.

Critical Weakness: Although a good set of baseline methods is used for experimental comparisons, the omission of VASA-3D is significant in my view. VASA-3D is a recent work that also leverages a 2D audio-driven generative model for generating 3D audio-driven Gaussian head avatars from a single image. VASA-3D produces compelling results that I consider stronger than the three baselines used in this submission. MVP4D and CAP4D are also recent relevant works given their multiview generation for 4D avatar construction, though not as necessary for comparison since they rely on pose/expression or 3DMM controls. To me, the results are not quite fully convincing without a VASA-3D comparison.

Question 1: Why is VASA-3D not included as a baseline? As mentioned above, the experimental comparisons would be more convincing to me if a clear advantage over VASA-3D were demonstrated. If there is a reason that such a comparison is not feasible, it should be explained.

Response:

We agree that VASA-3D is the most closely related recent work. Unfortunately, VASA-3D is not open-sourced, so it cannot be added as a baseline: the official page states that its models and APIs are not publicly available, the VASA-1 model it depends on is likewise unreleased, and we are not aware of a third-party reimplementation. Reimplementing it ourselves is not possible either, because VASA-3D is trained on videos synthesized by VASA-1 and driven by VASA-1 motion latents, so any substitute would no longer be VASA-3D. A direct comparison with VASA-3D is therefore not feasible. MVP4D is not open-sourced either. The released CAP4D takes FLAME expression and pose parameters rather than audio as its driving input; since its inputs differ from AudioLift's, we did not include it in our audio-driven protocol. To the best of our knowledge, besides the three original baselines (Real3D-Portrait, SDTalk, NeRFFaceSpeech), the only additional released audio-driven 3D method we could find whose inputs and outputs match AudioLift's (one image plus audio in, a 3D head renderable from free viewpoints out) is LAM (He et al., 2025), the released LAM-20K model with its official Audio2Expression module. We additionally evaluated LAM using the same reference portraits, audio segments, and scoring protocol, and provide the results in the following table:

DatasetMethodPSNR ↑SSIM ↑LPIPS ↓
RenderMe-360LAM14.080.6070.331
RenderMe-360AudioLift18.620.6780.216
HDTFLAM13.140.6340.352
HDTFAudioLift13.980.6530.335

AudioLift outperforms LAM on PSNR, SSIM, and LPIPS on both datasets. Note that LAM fits FLAME to the reference image and thus remains a parametric method with per-reference fitting.

We also clarify the difference between AudioLift and VASA-3D. VASA-3D optimizes a person-specific Gaussian avatar bound to FLAME for every identity, then drives it with VASA-1 motion latents, and does not model the back of the head. AudioLift uses no parametric face model and no per-identity optimization: Stage I generates five time-aligned views directly from one portrait and the audio, and Stage II reconstructs the Gaussians of every frame in one feed-forward pass. Additionally, AudioLift reconstructs the full head including the rear view, as shown in Figure 4 of the paper. We will include this discussion in the revised version.

I appreciate the authors' detailed rebuttal. I'm reading it carefully

Given the lack of publicly available code/models for VASA-3D, I agree that a controlled comparison should not be required. The clarifications on the differences between AudioLift and VASA-3D, specifically no per-identity optimization and no parametric face model, as well as reconstruction of the back of the head, help to better position this work. As VASA-3D is the most closely related work but hard to compare against empirically, I think a detailed discussion of these differences is essential to add to the paper.

Since a VASA-3D comparison is not included, I think the scope of empirical claims may need to be carefully recalibrated. The paper contains empirical claims relative to "existing methods", though these methods do not include VASA-3D.

Reviewer_wAexOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces AudioLift, a two-stage framework that generates a full-head 3D Gaussian talking avatar from a single portrait and an audio clip, without parametric face models or per-instance optimization. Stage I adapts an audio-conditioned video diffusion model (StableAvatar on a Wan2.1-1.3B backbone) to generate five synchronized views: four oblique views stitched into a 2x2 canvas plus a separate frontal stream, denoised jointly at a shared flow time so that global self-attention couples the views, with a clean reference latent prepended as a shared identity anchor and per-time audio windows injected through cross-attention. Stage II fine-tunes a feed-forward Gaussian reconstructor (FaceLift) to be robust to inconsistencies among the generated views, using view-aware corruption of the inputs (grid distortion and camera jitter on the non-frontal views, targets always clean), active-volume regularization that suppresses floaters outside a canonical head volume, and conservative rear-prior distillation from a frozen teacher across three rear yaw bins. Trained on NeRSemble speech recordings, the method is evaluated on RenderMe-360 and HDTF against three baselines under shared inputs, reporting frontal and novel-view PSNR/SSIM/LPIPS/FID, seven-view CSIM-ref, component ablations, and in-the-wild multilingual examples.

Critical strengths
  1. Compelling task and framing. Replacing the parametric motion medium (3DMM/FLAME) with generated multi-view video turns 4D avatar generation into a reconstruction problem; the single-portrait-plus-audio setting is practically valuable.
  2. Stage I is carefully engineered. Shared flow time across canvas/frontal streams, RoPE width-offset so global self-attention acts as cross-view attention, per-time audio windows, clean reference latent at a shared context index excluded from the loss; Fig. 22 supports frame-wise conditioning for degraded/off-frontal references.
  3. Stage II targets the actual failure mode. View-aware corruption (grid distortion + camera jitter, frontal kept clean, targets always clean) is a cheap, well-chosen proxy for generated-view disagreement; active-volume regularization addresses a known pixel-aligned-Gaussian pathology; rear distillation candidly compensates for NeRSemble's frontal-arc gap.
  4. Evaluation rigor well above average. Shared reference portrait and audio across methods, frame-0 exclusion to avoid anchor inflation, deterministic per-method canvas mappings, CSIM-ref detector-size sensitivity validated with IoU analysis, calibrated camera registration for novel-view scoring, identity-disjoint splits, and explicit statements of what was not done.
Critical weaknesses
  1. Closest prior art missing from experiments. VASA-3D (Xu et al., 2025) also produces audio-driven Gaussian head avatars from a single image and is cited but never compared or clearly differentiated; GaussianSpeech / LAM / MGGTalk likewise only cited. The core positioning claim is not empirically anchored against the nearest competitors.
  2. Two headline capabilities under-quantified. "Extreme expressions" rests on an auxiliary mouth-opening contrast on 16 curated in-the-wild clips (Table 4); "diverse accessories" is qualitative only. No expression/accessory metric on RenderMe-360/HDTF, which have the required GT.
  3. Novel-view fidelity evaluated at a single time step. All novel-view metrics are at frame 0; multi-view consistency over time is unmeasured, and the temporal consistency of the independently reconstructed per-frame Gaussian sequence is asserted rather than quantified.
  4. Long-horizon demonstration is qualitative. Fig. 1's 30s outputs and the window crossfade scheme have no quantitative support (no drift / identity-stability / boundary-quality metric).
  5. Frontal SSIM regression unexplained. Ours 0.678 trails Real3D-Portrait 0.700 and SDTalk 0.710 on RenderMe-360 despite leading PSNR/LPIPS/FID; the structural source of the gap is not analyzed.
Questions for authors
  1. How does AudioLift differ from VASA-3D (Xu et al., 2025), which also produces audio-driven Gaussian head avatars from a single image? Is VASA-3D parametric-free and optimization-free, and can it (or GaussianSpeech / LAM / MGGTalk) be added as a baseline? This is the comparison that would most anchor the paper's positioning claim.

  2. On RenderMe-360, your frontal SSIM (0.678) is below Real3D-Portrait (0.700) and SDTalk (0.710) despite better PSNR/LPIPS/FID. What explains this structural gap (high-frequency detail, motion blur, background handling), and which metric do you consider more relevant for avatar quality?

  3. Novel-view metrics are computed at frame 0 only. Can you report novel-view PSNR/SSIM/LPIPS over time (e.g., every k-th frame), and a temporal consistency measure of the Gaussian sequence (flicker or warping error between consecutive G_t)?

  4. What are the yaw/elevation ranges of the 16 NeRSemble training cameras versus the five RenderMe-360 target cameras, and is there angular overlap? Would the 12-camera holdout variant change the novel-view results?

  5. Table 3(a) shows a large jump from adding the frontal view (PSNR 16.19 to 20.18 under the internal protocol), and the 4-view number is below the external baselines. Is the gain mostly from having any near-frontal input rather than from five views per se, and does the same gap hold under the external 512x512 protocol?

5 comments
AuthorsOpenReview ↗

Response to Reviewer wAex (Part 1/4)

We sincerely thank you for your careful and insightful review. We greatly appreciate your recognition of the compelling task and framing, the carefully engineered Stage I, the Stage II design that targets the actual failure mode of generated-view disagreement, and the evaluation rigor. We are also grateful for your constructive comments and questions, which we address point by point below.

Critical Weakness 1: Closest prior art missing from experiments. VASA-3D (Xu et al., 2025) also produces audio-driven Gaussian head avatars from a single image and is cited but never compared or clearly differentiated; GaussianSpeech / LAM / MGGTalk likewise only cited. The core positioning claim is not empirically anchored against the nearest competitors.

Response:

AudioLift differs from VASA-3D in how the avatar is obtained and what it is bound to. Specifically, VASA-3D optimizes a person-specific Gaussian avatar bound to FLAME for every identity and is therefore neither parametric-free nor optimization-free, while AudioLift uses no parametric face model and no per-identity optimization. A matched-input comparison is not possible because the official VASA-3D page states that its models and APIs are not publicly available (project page, paper); likewise, GaussianSpeech's code is marked "coming soon" and MGGTalk has only a project page, so neither can be run.

We additionally evaluated LAM (the released LAM-20K with its official Audio2Expression module) using the same reference portraits, audio segments, and scoring protocol. AudioLift outperforms LAM on PSNR, SSIM, and LPIPS on both datasets.

DatasetMethodPSNR ↑SSIM ↑LPIPS ↓
RenderMe-360LAM14.080.6070.331
RenderMe-360AudioLift18.620.6780.216
HDTFLAM13.140.6340.352
HDTFAudioLift13.980.6530.335

Critical Weakness 2: Two headline capabilities under-quantified. "Extreme expressions" rests on an auxiliary mouth-opening contrast on 16 curated in-the-wild clips (Table 4); "diverse accessories" is qualitative only. No expression/accessory metric on RenderMe-360/HDTF, which have the required GT.

Response:

We define extreme expressions as large-amplitude changes in facial configuration, including wide mouth opening, strong smiles, and substantial eye or eyebrow movements (we select the top 25% of clips in each dataset by the range of the GT inner-lip opening over frames 1–80, i.e., the 90th minus the 10th percentile). We will clarify this in the revised version. All methods are evaluated on 11 RenderMe-360 clips from seven identities and six HDTF clips from six identities using frames 1–80. AudioLift achieves the best PSNR, LPIPS, FID, and CSIM-ref on the RenderMe-360 subset and the best results on all five metrics on the HDTF subset, supporting its reconstruction quality and identity preservation on these expressive examples.

For accessories, we select 13 identities and 26 clips from RenderMe-360 based on visible headwear or hair ornaments in the reference portraits; makeup alone does not qualify. AudioLift achieves the best PSNR, LPIPS, FID, and CSIM-ref on this fixed subset, demonstrating stronger reconstruction quality for subjects wearing accessories.

Extreme-expression subset — RenderMe-360

MethodPSNR ↑SSIM ↑LPIPS ↓FID ↓CSIM-ref ↑
AudioLift18.580.6770.21944.830.674
Real3D-Portrait17.850.6850.23782.200.644
SDTalk17.920.6990.24585.110.651
NeRFFaceSpeech11.350.4960.450100.560.377

Extreme-expression subset — HDTF

MethodPSNR ↑SSIM ↑LPIPS ↓FID ↓CSIM-ref ↑
AudioLift13.870.6260.35151.380.661
Real3D-Portrait13.610.6220.36756.070.647
SDTalk13.430.6210.36471.700.604
NeRFFaceSpeech10.720.5340.492128.690.402

Accessory subset — RenderMe-360

MethodPSNR ↑SSIM ↑LPIPS ↓FID ↓CSIM-ref ↑
AudioLift18.130.6430.23150.080.680
Real3D-Portrait16.550.6650.27997.090.632
SDTalk16.460.6770.26594.400.623
NeRFFaceSpeech11.200.4570.492126.070.407
AuthorsOpenReview ↗

Response to Reviewer wAex (Part 2/4)

Critical Weakness 3 (same as Question 3): Novel-view fidelity evaluated at a single time step. All novel-view metrics are at frame 0; multi-view consistency over time is unmeasured, and the temporal consistency of the independently reconstructed per-frame Gaussian sequence is asserted rather than quantified.

Response:

We provide (1) novel-view scores beyond frame 0 and (2) the warping error between consecutive frames of the per-frame Gaussian sequence in the following two tables.

Novel-view scores over time (42 RenderMe-360 clips, the five cameras and protocol of Table 1, every fifth frame; PSNR ↑ / SSIM ↑ / LPIPS ↓):

MethodFrame 0 (Table 1)Frames 5–80 (every 5th)
AudioLift12.27 / 0.582 / 0.44512.23 / 0.577 / 0.455
SDTalk10.99 / 0.571 / 0.47310.89 / 0.567 / 0.477
Real3D-Portrait10.82 / 0.557 / 0.49110.76 / 0.553 / 0.498
NeRFFaceSpeech8.51 / 0.426 / 0.6228.51 / 0.425 / 0.622

Warping error between consecutive frames (same five cameras, frames 1–80; frame t warped to t+1 with RAFT flow, mean absolute RGB difference (0–255) over the non-occluded head region; the real video is the reference level):

Real videoAudioLiftReal3D-PortraitSDTalkNeRFFaceSpeech
Warping error ↓2.051.822.360.730.82
Motion per frame (px)0.740.950.130.230.05

In the first table (novel-view scores over time), AudioLift remains first on all three metrics over the dynamic frames, and its per-frame means stay within 12.12–12.35 dB PSNR, 0.576–0.582 SSIM, and 0.445–0.459 LPIPS, i.e., within 1%, 0.8%, and 2% of their means. In the second table (warping error), AudioLift's error is below that of the real video while it moves as much as the real video, so the per-frame reconstruction does not introduce flicker. SDTalk and NeRFFaceSpeech score lower only because their heads are almost static at the novel cameras (0.23 and 0.05 px per frame), which fails to reproduce the head motion of the real video (0.74 px per frame); AudioLift therefore has the best warping error among the methods that reproduce the motion. We will include these results in the revised version.


Critical Weakness 4: Long-horizon demonstration is qualitative. Fig. 1's 30s outputs and the window crossfade scheme have no quantitative support (no drift / identity-stability / boundary-quality metric).

Response: We provide (1) the identity stability, (2) the frontal quality drift, and (3) the crossfade-boundary quality of the long outputs in the following three tables (21 HDTF identities, 10.9–42.6 s, Stage-II frontal renderings).

Identity over time (CSIM-ref ↑, ten relative-time bins, averaged equally across identities):

Time bin12345678910
CSIM-ref0.8350.8220.8260.8240.8170.8120.8210.8200.8220.812

Frontal quality over time (same ten relative-time bins, measured against the ground-truth frames with the protocol of the paper's Table 2; FID is computed per bin on about 1,700 frames):

Time bin12345678910
PSNR ↑14.2013.6813.7713.7713.4013.2813.7914.2114.2613.91
SSIM ↑0.6560.6440.6450.6480.6380.6350.6440.6520.6560.649
LPIPS ↓0.3310.3530.3480.3490.3660.3690.3500.3320.3280.346
FID ↓29.5829.7129.9631.1030.6530.3431.1231.5930.9230.18

Crossfade boundaries (mean absolute RGB change between adjacent frames, 0–1 scale):

At window boundariesInside windows
RGB change ↓0.00650.0058

The mean CSIM-ref remains within 0.812–0.835 across the bins, indicating stable identity similarity over time; the PSNR, SSIM, LPIPS, and FID also remain within narrow ranges (13.28–14.26 dB, 0.635–0.656, 0.328–0.369, and 29.58–31.59), indicating no quality drift; and the frame-to-frame change at the crossfade boundaries is close to that inside the windows, indicating boundary stability. We will include these results in the revised version.

AuthorsOpenReview ↗

Response to Reviewer wAex (Part 3/4)

Critical Weakness 5 (same as Question 2): Frontal SSIM regression unexplained. Ours 0.678 trails Real3D-Portrait 0.700 and SDTalk 0.710 on RenderMe-360 despite leading PSNR/LPIPS/FID; the structural source of the gap is not analyzed.

Response:

We think the SSIM gap partly comes from high-frequency detail inside the face region: our rendered faces have smoother facial detail than those of Real3D-Portrait and SDTalk. Specifically, SSIM compares local statistics in 7×7 windows and is sensitive to local texture and contrast, so this smoothness can reduce its score even when the generated face remains perceptually plausible.

However, for avatar quality we place great emphasis on FID, LPIPS, and novel-view metrics. FID measures whether the generated frames can be told apart from real video, which is the end goal of an avatar, and because it compares distributions rather than paired frames, it is not lowered when the generated performance is plausible but differs from the recorded one, which is the normal case in audio-driven generation. LPIPS is also an important proxy because it compares deep features and agrees with human judgments of whether two faces look alike, so it measures what an avatar is judged by in use. The novel-view metrics measure whether the avatar is correct as a 3D representation. AudioLift is best on all three in Table 1 and on FID and LPIPS in Table 2.


Question 1: How does AudioLift differ from VASA-3D (Xu et al., 2025), which also produces audio-driven Gaussian head avatars from a single image? Is VASA-3D parametric-free and optimization-free, and can it (or GaussianSpeech / LAM / MGGTalk) be added as a baseline? This is the comparison that would most anchor the paper's positioning claim.

Response 1:

Please see our response to Critical Weakness 1.


Question 2: On RenderMe-360, your frontal SSIM (0.678) is below Real3D-Portrait (0.700) and SDTalk (0.710) despite better PSNR/LPIPS/FID. What explains this structural gap (high-frequency detail, motion blur, background handling), and which metric do you consider more relevant for avatar quality?

Response 2:

Please see our response to Critical Weakness 5.


Question 3: Novel-view metrics are computed at frame 0 only. Can you report novel-view PSNR/SSIM/LPIPS over time (e.g., every k-th frame), and a temporal consistency measure of the Gaussian sequence (flicker or warping error between consecutive G_t)?

Response 3:

Please see our response to Critical Weakness 3.

Reviewer_BQ1cOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents AudioLift, a two-stage framework for generating full-head 3D Gaussian talking avatars from a single portrait image and audio without relying on parametric face models or per-person optimization. In Stage 1, the authors extend StableAvatar to jointly generate five synchronized audio driven videos using shared self-attention to encourage cross-view consistency. In Stage 2, a fine-tuned FaceLift reconstructor converts the generated multi-view frames into a sequence of 3D Gaussian avatars. To address inconsistencies among generated views and incomplete head geometry, the authors introduce view-aware corruption, active-volume regularization, and conservative rear-prior distillation. Experiments on RenderMe-360 and HDTF demonstrate improvements in several appearance and novel-view reconstruction metrics over existing 3D talking avatar methods, along with qualitative results showing expressive facial motion and full-head rendering.

Critical strengths
  • The paper presents a practical framework for generating full-head 3D talking avatars from a single image and audio without relying on parametric motion or per-person optimization. It achieves better quantitative performance than existing baselines and qualitatively improves the reconstruction quality of the back of the head and accessories.
Critical weaknesses
  • The proposed method primarily extends and combines existing models, StableAvatar and FaceLift, resulting in limited methodological novelty. Furthermore, the lack of direct comparisons with a straightforward combination of existing methods and sufficient ablation studies makes it difficult to assess the actual contributions of the proposed approach.
  • The quantitative evaluation of novel-view reconstruction is conducted only on frame 0, while quantitative evaluations of temporal consistency and audio-visual synchronization are lacking. Consequently, the core capabilities of the proposed audio-driven 4D avatar generation framework are not sufficiently validated.
Questions for authors
  • How does joint generation affect multi-view consistency compared to independently generating videos for each viewpoint?
  • How does the number of generated views in Stage 1 affect multi-view consistency and 3D reconstruction quality? Why do you first train a four-view model before extending it to five views, rather than training the five-view model from scratch?
  • What is the impact of removing the zero placeholder? Can you provide an ablation study to demonstrate its necessity?
  • Does interleaving in Stage 1 improve multi-view consistency?
  • Stage 1 adopts the spatial weighting strategy from the StableAvatar. How do you justify using the same weighting strategy in a multi-view setting, where the visibility of the face and lips varies across viewpoints?
  • Does Stage 2 preserve the audio-visual synchronization achieved in Stage 1? Can you provide quantitative results comparing synchronization before and after 3D reconstruction?
  • Why is the quantitative evaluation of novel-view reconstruction conducted only on frame 0, rather than on frames 1–80, where audio-driven facial motion occurs?
6 comments
AuthorsOpenReview ↗

Response to Reviewer BQ1c (Part 1/3)

We sincerely thank you for your thoughtful and detailed review. We greatly appreciate your recognition that AudioLift is a practical framework for generating full-head 3D talking avatars from a single image and audio without parametric motion or per-person optimization, that it achieves better quantitative performance than existing baselines, and that it qualitatively improves the reconstruction of the back of the head and accessories. We also thank you for your constructive questions and suggestions, which we address point by point below.

Critical Weakness 1: The proposed method primarily extends and combines existing models, StableAvatar and FaceLift, resulting in limited methodological novelty. Furthermore, the lack of direct comparisons with a straightforward combination of existing methods and sufficient ablation studies makes it difficult to assess the actual contributions of the proposed approach.

Response:

Main contribution: Our main contribution is a framework that replaces parametric motion control with conditional multi-view video generation for audio-driven 4D avatar generation. The audio-driven facial motion is synthesized directly as view-consistent multi-view video and then lifted to 3D by reconstruction; in this way, AudioLift achieves realistic expression control without a parametric face model or per-person optimization.

We build on StableAvatar and FaceLift because they provide pretrained video-generation and reconstruction priors, avoiding the cost of learning these capabilities from scratch. Beyond this, AudioLift has a fundamentally different goal and implementation from both. StableAvatar generates only a single-view portrait video; FaceLift performs static head reconstruction and has no temporal consistency when applied to video. Specifically, unlike the original single-view StableAvatar, our Stage I introduces architectural changes for joint multi-view video generation: a dedicated frontal token stream with coordinated positional and audio conditioning across views. Unlike the original FaceLift, we introduce view-aware corruption to handle inconsistencies among generated views, complemented by active-volume regularization and rear-prior distillation for compact and complete geometry. AudioLift is therefore not a straightforward combination of the two existing models.

Comparison with the straightforward combination of original methods: We provide the comparison with the straightforward combination of the original StableAvatar and the full released FaceLift pipeline, and with the released FaceLift reconstructor on identical Stage-I outputs, in the following table (eight NeRSemble development identities, 16 cameras, frames 1–80):

GenerationReconstructionPSNR ↑SSIM ↑LPIPS ↓
Original StableAvatarFull released FaceLift11.470.6440.374
Our Stage IReleased FaceLift reconstructor17.390.7260.276
Our Stage IOur Stage II17.920.7350.258

The complete AudioLift framework outperforms the direct combination by a large margin, and with the generated inputs held fixed, our reconstruction adaptation improves all three metrics. The component ablations in Section 4.3 of the paper further isolate the contribution of each design.

AuthorsOpenReview ↗

Response to Reviewer BQ1c (Part 2/3)

Critical Weakness 2: The quantitative evaluation of novel-view reconstruction is conducted only on frame 0, while quantitative evaluations of temporal consistency and audio-visual synchronization are lacking. Consequently, the core capabilities of the proposed audio-driven 4D avatar generation framework are not sufficiently validated.

Response:

We provide (1) novel-view scores beyond frame 0, which measure the temporal consistency of novel-view quality, and (2) audio-visual synchronization before and after reconstruction in the following two tables.

Novel-view scores over time (42 RenderMe-360 clips, the five cameras and protocol of Table 1, every fifth frame; PSNR ↑ / SSIM ↑ / LPIPS ↓):

MethodFrame 0 (Table 1)Frames 5–80 (every 5th)
AudioLift12.27 / 0.582 / 0.44512.23 / 0.577 / 0.455
SDTalk10.99 / 0.571 / 0.47310.89 / 0.567 / 0.477
Real3D-Portrait10.82 / 0.557 / 0.49110.76 / 0.553 / 0.498
NeRFFaceSpeech8.51 / 0.426 / 0.6228.51 / 0.425 / 0.622

Audio-visual synchronization (Stage II minus Stage I; 42 RenderMe-360 and 21 HDTF clips, full 81-frame sequences):

DatasetΔ Sync-C ↑Δ Sync-D ↓
RenderMe-360−0.0268−0.0466
HDTF−0.0376+0.0163

AudioLift remains first on all three metrics over the dynamic frames, and its per-frame means stay within 12.12–12.35 dB PSNR, 0.576–0.582 SSIM, and 0.445–0.459 LPIPS over the 17 sampled frames, i.e., within 1%, 0.8%, and 2% of their means, indicating temporal consistency of novel-view quality. Please see our response to Question 6 for the audio-visual synchronization, and our response to Question 7 for why the main table uses frame 0. We will include these results in the revised version.


Question 1: How does joint generation affect multi-view consistency compared to independently generating videos for each viewpoint?

Response 1:

The comparison between joint and independent generation is given in the following table (eight NeRSemble development identities, 16 cameras, frames 1–80):

GenerationReconstructionPSNR ↑SSIM ↑LPIPS ↓
Independent StableAvatarOur Stage II17.180.7280.288
Our joint Stage IOur Stage II17.920.7350.258

As shown in the table, joint generation outperforms independent generation on all three metrics. Specifically, joint generation uses shared self-attention as cross-view attention to allow each view to access expressions and facial information from other views at the same time step. This promotes multi-view consistency and provides compatible inputs for subsequent 3D reconstruction. Independent generation, however, lacks access to other views' visual information: although each independently generated view may look plausible on its own, even with the same audio their expressions may not correspond to the same facial state at a given moment, resulting in inconsistent views and conflicting observations for 3D reconstruction.

The independent row was generated with the original StableAvatar, one video per view, each starting from the frame-0 image of that view in our Stage-I output and driven by the same audio.


Question 2: How does the number of generated views in Stage 1 affect multi-view consistency and 3D reconstruction quality? Why do you first train a four-view model before extending it to five views, rather than training the five-view model from scratch?

Response 2

The ablation between four and five views is given in the following table (eight NeRSemble development identities, frames 1–80, generated videos and reconstruction model fixed):

Reconstruction inputsFrontal camera (paper Table 3(a)), PSNR / SSIM / LPIPSFour novel cameras, PSNR / SSIM / LPIPS
4 oblique views16.19 / 0.719 / 0.28416.59 / 0.717 / 0.298
4 oblique + frontal (5 views)20.18 / 0.779 / 0.18016.92 / 0.727 / 0.287

Five views outperform four on all three metrics at both the frontal camera and the four novel cameras, where the novel-camera gain indicates that the five generated views provide better multi-view consistency and 3D reconstruction quality.

We extended the trained four-view checkpoint rather than training five views from scratch because a four-view model was already trained at that point, and continuing from it reuses the multi-view representation learned in that stage, so that the subsequent training budget is devoted to incorporating the frontal stream instead of repeating the multi-view adaptation (Appendix C.4 and E). We will clarify this in the revised version.

AuthorsOpenReview ↗

Response to Reviewer BQ1c (Part 3/3)

Question 3: What is the impact of removing the zero placeholder? Can you provide an ablation study to demonstrate its necessity?

Response 3:

The zero placeholder is an engineering necessity of the token layout. The canvas stream carries the reference latent at t = 0, so the frontal stream needs a token at the same position to keep the two streams equal in length, which the inherited audio routing requires to split tokens into equal groups by time step (Appendix C.1). We fill it with zeros, so it is excluded from the loss and never decoded, adding no supervision or decoding cost. Zero-filling such conditioning positions is common practice in video diffusion models (Wan et al., 2025; Tu et al., 2025). Because removing the placeholder breaks the equal-length layout that the audio routing requires, an ablation without it is not applicable. We will clarify this in the revised version.


Question 4: Does interleaving in Stage 1 improve multi-view consistency?

Response 4:

Yes, interleaving supports multi-view consistency through audio alignment. Under the inherited audio routing, interleaving places the five views of one time step in one group, so that all five receive the audio of that time step. Without interleaving, the views would receive audio from different moments; for example, the frontal view would receive audio from the end of the clip, which would undermine multi-view consistency.


Question 5: Stage 1 adopts the spatial weighting strategy from the StableAvatar. How do you justify using the same weighting strategy in a multi-view setting, where the visibility of the face and lips varies across viewpoints?

Response 5:

We retain StableAvatar’s spatial weighting because multi-view consistency requires frontal and oblique views to depict the same speech articulation, making lip supervision important across views. We compute masks separately from each view’s training images, following its detected lip regions and foreground silhouette. Thus, even with the same weighting strategy, the weighted regions follow the viewpoint-dependent location, shape, and size of the lips, emphasizing each view’s articulation details and supporting consistent mouth motion across views.


Question 6: Does Stage 2 preserve the audio-visual synchronization achieved in Stage 1? Can you provide quantitative results comparing synchronization before and after 3D reconstruction?

Response 6:

Yes, Stage II preserves the audio-visual synchronization achieved in Stage I. We compared the Stage-I frontal video and Stage-II frontal rendering for the same clips and audio, reporting paired changes in Sync-C and Sync-D to measure the effect of reconstruction on synchronization. We evaluated 42 RenderMe-360 clips and 21 HDTF clips using complete 81-frame sequences. Differences are defined as Stage II minus Stage I.

DatasetΔ Sync-C ↑Δ Sync-D ↓
RenderMe-360−0.0268−0.0466
HDTF−0.0376+0.0163

Across both datasets, the mean Sync-C and Sync-D scores change by less than 1% of their respective Stage-I values, indicating that synchronization is preserved after reconstruction. This result is expected because Stage II is a reconstruction model: it reconstructs the facial states already generated by Stage I rather than generating a new performance. It retains the original frame-to-audio correspondence, helping preserve the synchronization established in Stage I.


Question 7: Why is the quantitative evaluation of novel-view reconstruction conducted only on frame 0, rather than on frames 1–80, where audio-driven facial motion occurs?

Response 7:

Frame 0 is the only time step at which all methods must reproduce the same expression: the reference portrait specifies it and the multi-camera GT records it. From frame 1 on, each method generates its own audio-driven performance, and the same speech admits many plausible expressions, so a method can produce a valid expression that differs from the one recorded in the GT. Therefore, when frames 1–80 are compared with the GT, this expression difference adds a penalty unrelated to reconstruction and lowers the scores, so on frames 1–80 these metrics cannot reliably show how well each method reconstructs novel views. This is why we did not use frames 1–80 for the cross-method novel-view comparison. Appendix Table 7 is consistent with this penalty: with identical generated inputs at the same four held-out cameras, every reconstructor scores lower on frames 1–80 than on frame 0 (e.g., PSNR 18.00 → 17.28 dB, LPIPS 0.241 → 0.264).

Nevertheless, we show that AudioLift remains the best-performing method in PSNR, SSIM, and LPIPS beyond frame 0; please see the first table of our response to Critical Weakness 2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.