acceptodds
Under review as a conference paper at ICLR 2027

Video Diffusion Avatars: Taming Video Diffusion Models into Controllable 3D Avatars

Abstract

Video diffusion models (VDMs) trained on internet-scale data encode a rich prior on appearance, motion, and physics that is difficult to replicate with explicit 3D rendering and costly 3D data capture. We introduce Video Diffusion Avatars (VDAs), a method that creates photo realistic, multi-view consistent avatars from a single image by fine-tuning a VDM for precise control over camera viewpoint, facial expression, and upper-body pose. Departing from 3D morphable models (3DMMs), VDAs use a learned expression space that lifts the expressiveness ceiling of 3DMMs, e.g. for the tongue and mouth interior. However, learned expression codes exhibit heavy viewpoint bias, as our DISTRACT benchmark reveals. We show that this bias can be removed through disentanglement, enabling their use as driving signals for multi-view consistent avatars. Camera and upper-body pose are controlled jointly via proxy-mesh renderings, giving independent control over expression, pose, and viewpoint. A camera-triplet training strategy makes the approach efficient: our models train for 34 hours on a single A100 using only public multi-view data. VDAs outperform state-of-the-art single-image 3D avatars on novel-view reenactment, reducing face LPIPS by 29% and improving perceptual quality (JOD) by 66% on BecomingLit. A static 3DGS reconstruction probe further confirms that our generations are the most multi-view consistent. Built on avideo prior, VDAs produce effects beyond the reach of explicit renderers, including physically plausible hair dynamics, shadows, reflections, and refractions.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Avatars, Controllable Video Diffusion, Facial Expressions, Camera Controlled Diffusion

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_b9VzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper adapts WAN-Animate into a single-image avatar model with controllable expression, pose, and viewpoint. It introduces DISTRACT to diagnose expression-code entanglement, combines learned expression conditioning with a pose-aware proxy mesh, and trains using multi-camera triplets. Reenactment benchmarks and a static 3DGS reconstruction probe demonstrate strong visual quality, quantitative performance, and multi-view consistency.

Critical strengths
  • The DISTRACT benchmark offers useful insights into viewpoint and identity entanglement in existing facial expression encoders.
  • The method achieves strong visual quality and state-of-the-art quantitative results on the evaluated reenactment benchmarks.
Critical weaknesses

Table 1 evaluates a separate MLP probe, but its improvements do not directly establish the effectiveness of the disentanglement modules used in VDA. The deployed modules are subsequently fine-tuned with only the flow-matching loss, leaving it unclear whether the invariance properties learned during disentanglement pretraining are preserved. The small gap between No disent. and Full in Table 5 also leaves the added benefit of explicit disentanglement unclear.

Please report DISTRACT results (viewpoint and identity invariance) for zexz_{\mathrm{ex}}, the re-entangled codes, and the PfaceP_{\mathrm{face}} outputs before and after VDA fine-tuning, with No disent. as a baseline where applicable. For zcamz_{\mathrm{cam}}, please evaluate whether samples from the same camera but different identities and expressions are closer than samples from different cameras. To clarify the role of pretraining, it would also help to compare the same architecture with (1) pretrained disentanglement modules kept frozen, (2) pretrained modules fine-tuned with VDA, as in the current method, and (3) modules trained jointly with VDA without separate pretraining.

Questions for authors

My questions are mainly about ablations in Table 5:

  • Could the authors provide 2 additional ablations, one replacing Ours (full) X-NeMo with WAN-Animate’s native expression encoder only, and one freezing the face encoder only? These ablations would help isolate the effects of encoder choice and face-encoder fine-tuning, as the existing comparisons change multiple components simultaneously.

  • How much do ray-direction coloring and separate head/torso coordinate frames contribute beyond the proxy mesh’s spatial cues? The no-proxy-mesh ablation does not isolate these effects. Two additional comparisons would help: (a) the same mesh rendered with Lambertian shading only, and (b) the same ray-direction coloring using a shared torso coordinate frame for both head and torso. These would isolate the benefits of directional coloring and part-specific coordinate frames, respectively.

3 comments

We thank the reviewer for the careful analysis and the precise questions. We address each point below.

Weaknesses

Tab. 1 and the DISTRACT benchmark merely serve as motivation, highlighting those expression spaces as promising candidates, if the view-bias is taken care of. We agree that these results are separate from the avatar use-case.

We now measure DISTRACT viewpoint invariance of the driving expression code at every stage of the pipeline, before and after joint fine-tuning. The stages are the raw X-NeMo code, the disentangled code, the code entering WAN-Animate's face encoder, and the face encoder's output tokens. We also report the camera-matching task described in Weakness 3 below, where a high score means camera information remains in the code. We perform the same measurement on the variant without disentangler, trained with camera triplets only:

StageOurs, beforeOurs, afterTriplets only, beforeTriplets only, after
Raw X-NeMo code17.217.217.217.2
After disentangler97.497.4––
Code entering the face encoder96.097.219.063.8
Face-encoder output tokens96.097.224.278.8
Camera matching on face-encoder tokens ↓0.80.242.66.2

For our model, the disentanglement is fully preserved.

We agree with the review, that there only is a small performance gap between Ours and "Ours without disentangler". The conclusion is that camera-triplet supervision alone makes the code inside the video model much more view-invariant (24% → 79%, as we reported in our supplementary (Figure 14)) and removes most camera information (43% → 6%). The cross-view training signal does most of the work by itself. Adding the disentangler still yields clearly stronger invariance at the code level (97% vs. 79%, with essentially no camera information left), and improved metrics, as shown in our orginal ablations.

Does zcamz_{cam} discriminate the view-point?

We designed an inverse version of DISTRACT to test exactly this. For each query image, the positive is an image from the same camera showing a different identity and expression. The four negatives come from other cameras, including the query's own identity and expression seen from a different camera. Rank-1 (%) is the share of queries for which the same-camera image is closest:

CodeSame camera, different identity and expression ↑Same camera, same identity, different expression ↑
Raw X-NeMo code58.271.8
Disentangled expression code0.42.0
Disentangled camera code88.493.6

Raw X-NeMo codes are heavily influenced by the camer view, resulting in 58% Rank-1 performance . The disentangled expression code removes this almost completely (0.4%), while the camera code captures it (88%).

The additional ablations and our answers to the questions follow in Part 2/2.

Additional Disentangler Ablations

As suggested, we trained Ours with a frozen disentangler, as well as with a disentangler trained jointly from random initialization:

VariantPSNR ↑LPIPS ↓JOD ↑ID ↑Expr. ↑AKD ↓APD ↓
Frozen Disentangler (pretrained)22.120.20904.6280.69580.91210.05754.16
Disentangler trained jointly (not pretrained)22.420.20954.6260.70090.91410.05324.18
Ours (pretrained, fine-tuned)22.430.20214.6380.69890.92010.05493.97
  • Frozen vs. fine-tuned. Fine-tuning the pretrained disentangler improves performance slightly.
  • From random initialization. For this version, we added the disentangler losses alongside the flow-matching criterion. Its performance is very close to that of the pretrained disentangler.

Question 1: Why we are using X-NeMo codes?

Our method is not tied to X-NeMo. We chose it because it showed the strongest separable expression information on DISTRACT (96% vs. 92.5%). The table below crosses the expression code with whether WAN-Animate's face encoder is trained:

Expression codeFace encoderPSNR ↑LPIPS ↓JOD ↑ID ↑Expr. ↑AKD ↓APD ↓
WAN-Animatefrozen21.520.20984.4380.70120.91150.05324.06
WAN-Animatetrained21.140.20154.6390.70420.92310.05364.11
X-NeMofrozen22.110.20874.6580.69770.92230.05493.96
X-NeMo (ours)trained22.430.20214.6380.69890.92010.05493.97

While X-NeMo tends to give higher PSNR, most metrics are roughly equal. The method therefore works with both latent spaces.

Question 2: Proxy-mesh shading style

We did not originally ablate this design, since we do not consider it a central contribution, but we agree that the ablation strengthens the paper. We trained three variants that change only how the proxy mesh is shaded: (i) Plücker coloring in a single coordinate frame (torso centric), (ii) Lambertian gray shading, and (iii) camera-space surface normals. We train Ours with these alternative proxy mesh styles and are evaluated on BecomingLit self-reenactment:

Proxy-mesh shadingPSNR ↑LPIPS ↓JOD ↑ID ↑Expr. ↑AKD ↓
Part-local Plücker (ours)22.430.2023.990.6990.9200.055
Global Plücker (torso frame)21.700.2043.940.7040.9140.055
Lambertian21.360.2043.960.7080.9230.056
Surface normals21.320.2043.950.6990.9230.053

The shading has only a small effect. We will add this ablation to the paper supplementary and reduce the emphasis on the part-local coloring accordingly.

Reviewer_b9VzOpenReview ↗

response to authors

Thank you for the additional experiments and clarifications. The evaluations of the latent codes and the controlled ablations address my concerns and strengthen the empirical support for the method. I maintain my weak-accept recommendation.

Reviewer_nPV5OpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents Video Diffusion Avatars (VDAs), which adapt a pretrained video diffusion model to generate portrait videos from a single reference image with independent control over facial expression, upper-body pose, and camera viewpoint. The method combines disentangled learned expression codes, a proxy mesh with part-local ray-direction coloring, and camera-triplet supervision on public multi-view datasets. Experiments demonstrate improved novel-view reenactment quality, while a static 3DGS reconstruction probe provides additional evidence of multi-view consistency.

Critical strengths
  1. An effective approach to controllable portrait generation. The combination of expressive learned motion codes and geometric conditioning addresses a practical challenge: preserving subtle expressions while independently controlling viewpoint and pose. The reported results show clear improvements in perceptual quality, identity preservation, and expression fidelity.

  2. Useful analysis of expression representations and training supervision. DISTRACT exposes viewpoint dependence in existing motion embeddings. Particularly interesting is the finding that camera-triplet supervision alone substantially improves viewpoint invariance. The ablations help distinguish this effect from the smaller additional benefit of the explicit disentangler.

  3. Accessible training with public data. Adapting the pretrained backbone in approximately 34 hours on a single A100 makes the training recipe practically attractive. Using public multi-view datasets also supports reproducibility.

  4. Substantial experimental validation. The paper evaluates multiple held-out datasets, includes relevant generative and explicit-avatar baselines, and provides extensive ablations. The static reconstruction probe complements image-level metrics, and the streaming prototype offers a useful preliminary investigation of inference efficiency.

Critical weaknesses
  1. Further isolate the benefit of the proposed proxy-mesh encoding. Removing the entire proxy mesh demonstrates the importance of dense geometric conditioning, but does not isolate the contribution of part-local ray-direction coloring. Comparisons against normal-map conditioning, world-frame ray directions, and a shared coordinate frame for the head and torso would clarify which aspects of the proposed encoding improve camera–neck-pose disentanglement.

  2. Extend consistency evaluation to dynamic and independently generated sequences. The static 3DGS probe is a useful test, but its results also depend on image fidelity and reconstruction quality. An additional experiment could generate the same expression sequence along different camera trajectories and compare outputs at matching times and viewpoints. This would strengthen the evidence that consistency extends across separate generations and remains reliable during facial motion, rather than only under a frozen expression.

  3. Further characterize expression fidelity beyond the current metrics. The paper appropriately evaluates expressions in an independent DECA space. However, FLAME-based measurements may not fully capture the tongue, mouth-interior details, and subtle expressions that motivate learned expression control. A targeted evaluation of these cases, potentially with human judgments, would better demonstrate this advantage. Similarly, the authors clearly identify the DISTRACT MLP probe as transductive; reporting probe performance on held-out identities or sequences would provide complementary evidence of generalization.

Questions for authors

The paper presents a well-motivated and effective combination of expression representations, geometric conditioning, and cross-view supervision. I find the camera-triplet training analysis particularly useful, and the reported improvements across multiple evaluation settings support the practical value of the approach. Its contribution lies in a carefully validated adaptation of video diffusion models for controllable avatars. The suggestions above would sharpen the attribution of individual design choices and strengthen the characterization of consistency and expressiveness.

3 comments

We thank the reviewer for the thorough review and the many concrete suggestions. We address each point below.

Weakness 1: Missing ablation of the part-local ray (Plücker) conditioning

We did not originally ablate this design, since we do not consider it a central contribution, but we agree that the ablation strengthens the paper. We trained three variants that change only how the proxy mesh is shaded: (i) Plücker coloring in a single coordinate frame (torso centric), (ii) Lambertian gray shading, and (iii) camera-space surface normals. We train Ours with these alternative proxy mesh styles and are evaluated on BecomingLit self-reenactment:

Proxy-mesh shadingPSNR ↑LPIPS ↓JOD ↑ID ↑Expr. ↑AKD ↓
Part-local Plücker (ours)22.430.2023.990.6990.9200.055
Global Plücker (torso frame)21.700.2043.940.7040.9140.055
Lambertian21.360.2043.960.7080.9230.056
Surface normals21.320.2043.950.6990.9230.053

The shading has only a small effect, mainly on PSNR. We will add this ablation to the paper supplementary and reduce the emphasis on the part-local coloring accordingly.

Weakness 2: Extending the 3DGS probe to a dynamic 4D representation

We chose a static 3DGS probe because it isolates multi-view consistency from the open problems of dynamic avatars. 3DGS cannot model complex lighting effects, faithfully representing facial motion and the mouth interior is a research problem of its own, and the optimization dynamics of 4D fitting are far from trivial. Thus fitting failures would confound the measurement. Nevertheless, we agree that this additional experiment strengthens our paper.

We can share preliminary results on 3 of 25 BecomingLit examples. Here, we compare against CAP4D and use their codebase. We only exchange their diffusion-based image generator with Ours.

Generated training dataPSNR ↑LPIPS ↓JOD ↑ID ↑AKD ↓
CAP4D21.080.3532.540.6040.0602
Ours21.250.2812.550.7100.0570

Avatars fitted to our generations are clearly better in perceptual quality, identity, and keypoint accuracy. We will extend this experiment to all identities and report the results in the final version.

Weakness 3.2: Generalization of the MLP probe

The reviewer is right, and we apologize for this oversight. The probe was one of our earliest internal experiments and was trained on embeddings that included the benchmark frames. Thus, we have properly repeated the experiment with the benchmark frames and identities excluded from training. We then evaluate on the unchanged benchmark:

TaskEncoderRaw codesMLP Probe (submitted, faulty)MLP Probe (updated)
View inv.DECA618579.1
View inv.PD-FGC89086.7
View inv.WAN-Animate219492.5
View inv.X-NeMo189896.3
Identity inv.DECA343240.7
Identity inv.PD-FGC74648.9
Identity inv.WAN-Animate146668.5
Identity inv.X-NeMo147676.7

The held-out results closely reproduce the original finding, without changing the conclusions.

We also agree with the reviewer that the probe says nothing directly about the avatar model. Its role in the paper is a sanity check that motivates our design: it shows that the frozen codes contain the expression information needed to tell expressions apart, entangled with viewpoint rather than lost. We will replace the probe experiments, and update the text and numbers accordingly.

Weakness 3.1 (expression similarity) follows in Part 2/2.

(Part 2/2)

Weakness 3.1: Expression similarity is reported with DECA only

We now report expression similarity with four encoders. Two 3DMM regressors (DECA and EMOCA) and the two strongest encoders on DISTRACT (X-NeMo and WAN-Animate). Each cell is the cosine similarity between the expression codes of the generated and the ground-truth frames (higher is better). Note that cosine similarity can behave vastly different in different latent spaces

Self-reenactment (BecomingLit).

MethodDECAEMOCAX-NeMoWAN-A.
LAM0.6700.7660.9630.839
UIKA0.8650.8470.9780.858
FlexAvatar0.8900.8780.9820.875
CAP4D0.8520.8800.9810.885
MMDM0.8530.8750.9810.916
MVP4D0.8310.8650.9780.923
Ours0.9200.9110.9880.958

Cross-reenactment (BecomingLit).

MethodDECAEMOCAX-NeMoWAN-A.
LAM0.6230.6400.9520.812
UIKA0.7750.7310.9740.849
FlexAvatar0.7000.6160.9530.788
CAP4D0.7700.7750.9780.871
MMDM0.7800.7630.9780.907
MVP4D0.7550.7500.9760.917
Ours0.8080.7930.9820.950

Our method ranks first on all four encoders, on both self- and cross-reenactment. X-NeMo and WAN-Animate are the code spaces our model is conditioned on, so these two columns may favor our method. DECA and EMOCA are independent of our pipeline and show the same ranking.

Ablations (self-reenactment, BecomingLit). The same four encoders on the rows of our ablation table:

VariantDECAEMOCAX-NeMoWAN-A.
WAN-A. (training-free)0.82790.85270.97550.9493
WAN-A. disent. (training-free)0.83860.85890.97900.9516
Ours, WAN-A. expr.0.91150.90060.98630.9639
Ours, WAN-A. expr. + frozen fusers0.81750.86090.97550.9297
Ours, no disent.0.90990.90700.98800.9575
Ours, no disent., self-driven0.87440.89610.98490.9423
Ours, no expr. codes0.87960.87660.98120.9511
Ours, no proxy mesh0.91240.89090.98540.9538
Ours, rand. corpus0.89540.88150.98480.9114
Ours (full)0.92010.91110.98780.9578

On the two independent encoders, DECA and EMOCA, the full model scores highest of all variants. On X-NeMo and WAN-Animate, the variant conditioned on that same code space matches or slightly exceeds it, as expected when the metric is computed in the conditioning space.

We additionally ran a VLM-based expression similarity ranking using Claude Opus 5.5. The VLM was tasked to rank all seven methods by expression similarity to the ground truth. Faces are masked onto a uniform background and shuffled per grid, so methods cannot be identified. Our method has the best mean rank (1.90 self / 2.02 cross; next best 3.52 / 3.20). While the VLM is not perfectly reliable in ranking expressions, especially when it comes to tiny details, the gap comapared to the second best model clearly supports VDA.

MethodSelf: mean rank ↓Self: top-1 ↑Cross: mean rank ↓Cross: top-1 ↑
LAM6.532%6.200%
UIKA4.885%4.388%
FlexAvatar3.528%5.452%
CAP4D3.653%3.428%
MMDM3.5713%3.2012%
MVP4D3.955%3.3312%
Ours1.9063%2.0257%

Thank you for your detailed response and the additional results. My concerns have been adequately addressed, and I have therefore raised my score from 3 to 4.

Reviewer_kFQ5OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents a method for generating controllable human avatar videos from a single input image using a pretrained video diffusion model. The method enables separate control over facial expression, upper body pose, and camera viewpoint. The method disentengles expression from viewpoint in the driving features extracted by a pretrained encoder. Camera viewpoint and upper body are controlled through proxy meshes. The model is fine-tuned using camera triplets, where the reference image, driving video, and target video come from different viewpoints. Experiments show improved novel-view generation performance over the baselines, while a static 3DGS reconstruction probe supports multi-view consistency

Critical strengths
  • Clear motivation and methodology. The motivation is well articulated, and the proposed method is clearly presented and easy to follow.

  • Addressing viewpoint bias in expression control. The paper highlights the importance of disentangling facial expression from viewpoint for controllable avatar generation and proposes a concrete approach to address this issue.

  • Strong quantitative and qualitative results. The proposed method achieves strong quantitative results against the evaluated baselines. The provided videos also demonstrate convincing visual quality and multi-view consistency.

  • Accessible training setup. The method builds on publicly available datasets and pretrained checkpoints, with fine-tuning completed in approximately 34 hours on a single A100 GPU. This accessible setup could facilitate further research and make the method a useful baseline for controllable human avatar generation.

Critical weaknesses
  • Limited evidence of disentanglement generalization. Table 1’s MLP probe is trained on embeddings that include benchmark frames, so it does not establish generalization to unseen data. Furthermore, improved expression retrieval alone does not demonstrate the removal of viewpoint information.
  • Insufficient validation of the disentanglement objective. Table 5 ablates the disentangler as a whole but does not isolate its individual losses. Sensitivity to loss weights and cycle-loss scheduling is also unreported, leaving the necessity and robustness of these choices unclear.
  • Limited analysis of failure cases. While the paper discusses limitations, representative failure videos and an analysis of their causes would help readers better understand when the method fails and its practical limitations.
Questions for authors
  • Under what conditions does explicit disentanglement provide clear benefits over camera-triplet training alone, particularly for large viewpoint changes or challenging expressions?
2 comments

We thank the reviewer for the careful reading and the constructive criticism. We address each point below.

Weakness 1: Generalization of the MLP probe

The reviewer is right, and we apologize for this oversight. The probe was one of our earliest internal experiments and was trained on embeddings that included the benchmark frames. Thus, we have properly repeated the experiment with the benchmark frames and identities excluded from training.

TaskEncoderRaw codesMLP Probe (submitted, faulty)MLP Probe (updated)
View inv.DECA618579.1
View inv.PD-FGC89086.7
View inv.WAN-Animate219492.5
View inv.X-NeMo189896.3
Identity inv.DECA343240.7
Identity inv.PD-FGC74648.9
Identity inv.WAN-Animate146668.5
Identity inv.X-NeMo147676.7

The held-out results closely reproduce the original finding, without changing the conclusions.

We also agree with the reviewer that the probe says nothing directly about the avatar model. Its role in the paper is a sanity check that motivates our design by showing that the frozen codes contain the expression information needed to tell expressions apart. We will replace the probe experiments, and update the text and numbers accordingly.

Weakness 2: Missing ablation of the disentangler training

We agree and have added this ablation. Each variant is trained on the same data as our disentangler. It is evaluated on DISTRACT viewpoint and identity invariance.

Disentangler training lossesView inv. ↑Identity inv. ↑
Reconstruction + direct pairwise supervision (no contrastive, no cycle)57.755.0
Reconstruction + contrastive, no cycle98.381.2
Full, cycle losses from the first epoch97.882.5
Full (ours)98.283.1

The supervised contrastive loss is crucial. Replacing it with direct regression-style supervision of the code pairs drops viewpoint invariance from 98% to 58%. The cycle loss helps identity invariance slightly.

Weakness 3: Failure cases

VDA is very robust to challenging reference images, but the transfer of the eye gaze and tongue movement sometimes fails. Extreme asymmetric expressions sometimes pose an issue as well. Furthermore, fast camera movement can sometimes result in grid-like artifacts. The updated PDF will include a failure case figure in the supplementary material.

Question: Disentangler vs. camera-triplet supervision alone?

The reviewer's observation matches ours: camera-triplet supervision alone already yields substantial viewpoint invariance. The table below shows the relevant rows of our self-reenactment ablation:

VariantPSNR ↑LPIPS ↓JOD ↑ID ↑Expr. ↑AKD ↓APD ↓
No disentangler, self-driven (no camera triplets)20.690.22074.6130.68660.87440.06084.51
No disentangler, camera triplets22.060.20484.6080.69270.90990.05503.94
Ours (disentangler + camera triplets)22.430.20214.6380.69890.92010.05493.97

Camera-triplet supervision is the main driver of self-reenactment quality. It is worth 1.38 dB PSNR and clearly better expression, keypoint and pose accuracy. The disentangler adds a smaller improvement on top, mainly in image quality and in fine details. In particular, the camera-triplet model can handle large view-point changes well, as it is trained for such a use-case.

Thank you for the detailed response. My concerns have been addressed, and I will maintain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.