acceptodds
Under review as a conference paper at ICLR 2027

UniPortrait4D: Unifying 4D Portrait Reshooting and Animation

13 views

Abstract

Portrait animation and reshooting (novel-view synthesis) have been extensively studied for video generation, yet existing approaches typically treat them as separate problems: expression-driven animation is often restricted to a fixed viewpoint, while reshooting relies on limited multi-view training data with insufficient identity diversity. We present UniPortrait4D, a unified video DiT that supports both synchronized multi-view reshooting from a monocular portrait video and 4D portrait animation. To unify 4D portrait reshooting and animation, we leverage FLAME-based rendering as a unified camera representation for multi-view generation. For reshooting, we additionally incorporate point-cloud renderings to improve multi-view consistency. For animation, we employ a high-capacity latent representation for facial expressions and adversarially train it with gradient reversal to reduce driver-identity predictability while preserving fine-grained facial motion. Benefiting from the 4D consistency of our video DiT, the generated multi-view videos can be reconstructed into a 4DGS representation, enabling real-time free-viewpoint rendering. Extensive experiments demonstrate that our method achieves strong reference-identity preservation in cross-identity animation and high multi-view consistency in reshooting.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
portrait animation, novel-view synthesis, video diffusion models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_uEt1OpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents UniPortrait4D, a unified video diffusion transformer for portrait reshooting and cross-identity animation. For reshooting, the method combines point-cloud renderings and FLAME geometry with view attention to improve multi-view consistency. For animation, it employs a high-dimensional expression encoder with gradient-reversal-based adversarial training to suppress driving-identity information while preserving facial expressions. The generated multi-view videos are further used for 4D Gaussian Splatting reconstruction, enabling free-viewpoint rendering. Experiments on AVA-256 and FEED demonstrate improvements over existing reshooting and animation baselines in terms of generation fidelity, multi-view consistency, identity preservation, and motion transfer. Ablation studies further evaluate the effectiveness of the proposed components and unified training.

Critical strengths

1. Well-motivated problem formulation and unified framework. The paper identifies a meaningful research gap by recognizing the complementary characteristics of portrait reshooting and animation. While reshooting benefits from multi-view supervision but suffers from limited identity diversity, animation can leverage diverse identities and facial motions from monocular videos. The proposed unified framework is a well-motivated approach to exploiting these complementary sources of supervision. Furthermore, the hybrid geometry conditioning and identity-adversarial expression encoding are thoughtfully designed to address the respective challenges of the two tasks.

2. Strong empirical performance and comprehensive evaluation. The proposed method demonstrates strong performance compared to existing baselines on both portrait reshooting and animation tasks. In particular, the evaluation considers not only the fidelity of generated videos but also downstream 4DGS reconstruction quality and multi-view consistency, which are well aligned with the paper's objectives. Additionally, the ablation studies provide empirical evidence supporting the effectiveness of the proposed hybrid geometry conditioning and identity-adversarial training.

Critical weaknesses
  1. Insufficient details on unified training. Although unified training is a central contribution, the paper does not clearly specify the training objective, task sampling strategy, or how the reshooting and animation tasks are jointly optimized. These missing details make the proposed framework difficult to fully understand and reproduce.

  2. Insufficient validation of unified training. Table 3(b) only demonstrates that reshooting supervision benefits animation. The reverse direction, whether animation training improves reshooting performance, is not evaluated, leaving the claimed mutual benefits insufficiently supported.

  3. Lack of clarity in methodology and presentation. Several notations (e.g., IdI_d and zt,vz_{t,v}) are introduced without clear definitions. Moreover, Figure 3 lacks sufficient explanation in the main text, making it difficult to understand the roles of its components and the overall training pipeline.

Questions for authors
  1. Clarification of unified training. Could the authors provide more details on the unified training procedure, including the training objective, task sampling strategy, and how the two tasks are jointly optimized? Additionally, is the pretrained expression encoder frozen or jointly fine-tuned during unified training?

  2. Ablation on unified training. Table 3(b) demonstrates that reshooting supervision benefits animation. However, the reverse direction is not evaluated. Could the authors provide a comparison between reshooting-only and unified training to verify whether animation supervision also improves reshooting performance?

  3. Evaluation of multi-view animation. While Table 3(b) provides some evidence of improved multi-view consistency, the evaluation of 4D animation remains limited. Could the authors provide quantitative comparisons with relevant prior methods, such as MVP4D, GeoDiff4D, and FactorPortrait, where applicable? In particular, comparisons of multi-view consistency and downstream 4D avatar reconstruction quality would help substantiate the claimed 4D animation capability.

4 comments
AuthorsOpenReview ↗

Response to Reviewer uEt1

We thank the reviewer for the valuable feedback on the training procedure, task interaction, notation, and 4D animation evaluation. Additional experiments are in Appendix D, and the notation and training-sampling strategy have been clarified in the revised manuscript.

1. Unified training objective and sampling

Training consists of three stages. First, we train the pose/expression encoding and appearance reconstruction framework through DeX-Portrait-style self-supervised reconstruction (Shi et al., 2026). Second, we freeze the expression encoder and train an identity probe, then jointly optimize the encoder, probe, and reconstruction pathway through gradient reversal while retaining the reconstruction objective. Third, we freeze the resulting expression encoder and train the shared DiT with the standard flow-matching objective.

During DiT training, reshooting, monocular animation, and multi-view animation are sampled at a 5:2:3 ratio. Reshooting uses synchronized NeRSemble videos from different cameras, with one view as the source and target-camera videos as ground truth. Monocular animation uses a same-identity reference image and driving clip, supervised by the driving video. Multi-view animation uses a same-identity reference and synchronized clips; the driving camera may differ from the target cameras, and one to four target views are sampled. All modes use flow matching against their ground-truth target videos. The stage-wise optimization, sampling, and supervision are documented in Appendix B.1, Three-Stage Training and Task Sampling (pp. 15–16).

2. Does animation training also benefit reshooting

Yes. Figure 9(a) (Appendix D.1, p. 18) shows improved reshooting on challenging in-the-wild videos, including exaggerated expressions, tongue/eye/mouth details, and source illumination. At the data level, animation introduces abundant monocular videos with more varied expressions and lighting. At the architecture level, source-video expression latents are also injected through cross-attention during reshooting, providing additional guidance for complex expressions. Together, these provide a route for animation training to improve reshooting generalization.

3. Notation and Figure 3

IdI_{d} denotes the driving image supplied to the expression encoder FϕF_{\phi}. zt,vz_{t,v} denotes the noisy video latent at video time index tt and target-view index vv. We have added these definitions to Sections 3.3 and 3.2, respectively. We have also expanded the self-supervised initialization paragraph in Section 3.3 to explain the appearance encoder, pose-based feature warping, expression encoder, and reconstruction decoder, with an explicit reference to Figure 3; architectural details are provided in Appendix B. The added notation definitions and initialization explanation are highlighted in blue on pp. 4–5. These revisions distinguish motion-encoder initialization from GRL post-training and the subsequent DiT training.

4. Comparisons with 4D animation methods

We could not obtain publicly released implementations of GeoDiff4D (Xu et al., 2026) and FactorPortrait (Tang et al., 2025) for this revision. We therefore compare with MVP4D (Taubner et al., 2025a), UIKA (Wu et al., 2026), and CAP4D (Taubner et al., 2025b). The added evaluation uses 50 identities selected from AVA-256 and reports cross-identity reenactment only. Qualitative results compare reconstructed 4DGS avatar renderings in Figure 14; quantitative results are given in Table 4 (Appendix D.5, p. 21):

MethodCSIM ↑AED ↓APD ↓MET3R ↓
UIKA0.62400.02880.43590.3184
CAP4D0.67970.03290.46770.3726
MVP4D0.57320.03280.40180.3705
Ours0.69550.02590.31060.2546

Our method achieves the highest CSIM and lowest AED, APD, and MET3R among these baselines. MET3R (Asim et al., 2025) is measured between synchronized frames from the generated multi-view videos, not from the reconstructed avatar renderings; lower is better. View attention supports cross-view information exchange, improving consistency and reducing reconstruction artifacts, while GRL reduces driver-identity leakage and improves reference-identity preservation. The qualitative examples and quantitative results support these advantages.

Reviewer_uEt1OpenReview ↗

Official comment by reviewer uEt1

Thank you for the response and additional qualitative results. While the explanation and Figure 9(a) are helpful, I believe my concern regarding the mutual benefits of unified training remains insufficiently addressed.

My original question specifically requested a controlled ablation comparing reshooting-only training and unified training to verify whether animation supervision improves reshooting performance. Although the additional qualitative results suggest potential improvements, they do not clearly isolate the contribution of animation training.

I would appreciate a quantitative comparison under comparable training settings to substantiate this claim. Otherwise, I believe the claim regarding the bidirectional benefits of unified training should be stated more cautiously.

AuthorsOpenReview ↗

Response to Reviewer uEt1

Thank you for the suggestion. We provide below a quantitative comparison between reshooting-only and unified training. We evaluate on 50 test identities from AVA-256, none of which appear in the training data. We use the same evaluation metrics as Table 1 in the main paper's Comparison section: PSNR, SSIM, and LPIPS for Generated-to-GT, 4DGS-to-GT, and generated-view consistency (GVC).

Both models use the same architecture. The reshooting-only model is trained for 40,000 iterations using only NeRSemble (3D data). The unified model is trained for 40,000 iterations (20,000 animation iterations and 20,000 reshooting iterations) using VFHQ (2D data) and NeRSemble (3D data). We find that the unified model outperforms the reshooting-only model under comparable training settings, providing evidence that animation supervision improves reshooting performance.

TrainingGen-to-GT4DGS-to-GTGVC
PSNR ↑SSIM ↑LPIPS ↓PSNR ↑SSIM ↑LPIPS ↓PSNR ↑SSIM ↑LPIPS ↓
Reshooting-only19.510.73320.284119.130.70510.317123.690.76580.2453
Unified21.400.77400.266420.760.75890.279624.330.78720.1970

Unified training improves all three groups of metrics. We believe that the additional animation data, covering more diverse expressions and lighting conditions, improve reshooting generalization to in-the-wild inputs, yielding clearer facial details, as illustrated in Figure 9(a) (p. 19). In Figure 9(a), “w/o Animation” denotes the reshooting-only results, while “Ours” denotes the results of unified training. We have also added this quantitative comparison to Appendix D.1 as Table 4 (p. 20).

Reviewer_uEt1OpenReview ↗

Official comment by reviewer uEt1

Thank you for your detailed response. While this topic is somewhat outside my primary area of expertise (as reflected in my confidence rating), I found your clarifications helpful and believe they adequately address my concerns. I have therefore decided to raise my score.

Reviewer_LW8pOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

UniPortrait4D unifies portrait reshooting and animation based on a shared video diffusion transformer. The reshooting branch integrates hybrid geometric conditions of point cloud rendering and FLAME global normals, and combines a view attention mechanism to improve multi-view spatiotemporal consistency; the generated multi-view videos can be reconstructed into a 4D Gaussian representation. The animation branch adopts high-dimensional expression latents and conducts identity adversarial training via a gradient reversal layer, which suppresses identity leakage while preserving fine-grained facial motion. Joint training of the two tasks achieves mutual performance gains. The method outperforms existing baseline methods on both tasks, and ablation studies validate the effectiveness of each module design.

Critical strengths
  1. The approach of introducing a gradient reversal layer (GRL) for adversarial training between the expression encoder and identity encoder is, in my view, rather inspiring for the research community.
  2. The demo results in the supplementary materials appear stable and natural, demonstrating the practical effectiveness of the proposed method.
  3. This paper is well organized and easy to understand.
Critical weaknesses
  1. Several baselines are missing. For the reshooting direction, works such as 4DAnyone (Jin et al., 2026) and Scaling4D (Cai et al., 2026) are not included. 4DAnyone is mentioned in the metrics section, but I am wondering why it is not adopted as one of the baseline methods. For portrait animation, works including LivePortrait (Guo et al., 2024) and DeX-Portrait (Shi et al., 2026) are also absent. Notably, DeX-Portrait is used for self-supervised motion encoder initialization in the paper, so it is unreasonable that it does not appear in the comparative experiments.
  2. The general novelty is limited. I acknowledge the engineering contribution of integrating multiple 4D tasks (reshooting and animation) into a single DiT model, as well as the experimental contribution of introducing 4DGS for further downstream tasks. However, compared to the level of innovation expected for an academic paper, this work reads more like a report of a relatively complex engineering system.
  3. There is a lack of discussion and comparison with pure 2D methods. Currently, large-scale pre-trained DiTs (whether open-source or closed-source) already support universal reference-based generation, such as MiniMax-H3-Ref2VA and Seedance 2.0/2.5. It remains unclear whether these models can perform reshooting and animation tasks in a zero-shot manner, and what the advantages and disadvantages of such methods are compared to the approach in this paper.
  4. The quantitative metrics of the proposed method are close to those of Wan-animate, making it difficult to demonstrate a significant advantage of the method.
Questions for authors
  1. Add the baselines mentioned in the weaknesses section.
  2. Add discussion and comparison with pure 2D methods.
  3. Explain novelties in this paper.
2 comments
AuthorsOpenReview ↗

Response to Reviewer LW8p

We thank the reviewer for the valuable comments and suggestions. Additional experiments are in Appendix D; the new qualitative and quantitative baseline comparisons are included in the main paper.

1. Missing baselines

Video reshooting. 4DAnyone (Jin et al., 2026). We have added a comparison with 4DAnyone, which uses skeleton renderings to condition camera control. Skeleton estimation from tightly cropped portraits can be unreliable, as shown in Figure 11(a) (Appendix D.3, p. 19). These estimation errors can lead to failed generations. We therefore run 4DAnyone on full-body source videos and crop the facial regions of its reconstructed 4DGS renderings for comparison; our method receives face crops of the same source videos. Our method achieves better facial consistency and detail preservation in the compared examples (Figure 11(b), Appendix D.3, p. 19).

Scaling4D (Cai et al., 2026). Its code has not been publicly released, so we could not include a comparison in this revision. We will add a comparison when a public implementation becomes available.

Portrait animation. We have added LivePortrait (Guo et al., 2024) and DeX-Portrait (Shi et al., 2026). Our GRL-based training (Ganin et al., 2016) reduces driver-identity leakage and improves identity preservation, yielding higher CSIM than DeX-Portrait. Our camera and pose control based on FLAME (Li et al., 2017) provides accurate pose alignment, reflected in substantially lower APD than LivePortrait. The updated results from Table 2 are reproduced below; qualitative comparisons are shown in Figure 5 (p. 7).

MethodPSNR ↑SSIM ↑LPIPS ↓CSIM ↑APD ↓AED ↓
LivePortrait25.080.8110.1520.6192.3690.0294
DeX-Portrait25.630.8170.1140.6021.6970.0256
Ours27.200.8410.1070.6301.3200.0241

2. Novelty through mutually beneficial training and identity adversarial motion encoding

Mutually beneficial joint training. Unifying animation and reshooting in a single DiT yields bidirectional gains rather than simply combining two task interfaces (Appendix D.1, Figure 9, p. 18).

Animation benefits reshooting at both the data and architecture levels. At the data level, animation training incorporates large-scale monocular videos, exposing the shared renderer to a broader range of expressions and lighting conditions than the limited multi-view reshooting datasets. This improves its generalization to challenging in-the-wild inputs. At the architecture level, expression conditioning is also used during reshooting: we extract expression latents from the source video and inject them into the shared DiT through expression cross-attention, providing additional guidance for preserving complex facial motion. Figure 9(a) shows that joint training improves tongue, eye, and mouth details, as well as preservation of the source lighting.

Conversely, reshooting supervision encourages the shared renderer to learn correspondences between tokens across viewpoints, improving the cross-view consistency of multi-view animation, particularly eye and mouth alignment (Figure 9(b)). These bidirectional benefits constitute a contribution of our unified framework.

GRL for identity suppression. A second contribution is identity-adversarial post-training of a high-capacity expression encoder. GRL suppresses decodable driver-identity cues without imposing a low-dimensional bottleneck, improving identity preservation while retaining detailed motion and comparable pose and expression accuracy (Table 3(a); Figure 6(a)).

3. Comparisons with MiniMax H3 and Seedance

Please see Appendix D.4 (p. 20) for our comparisons with MiniMax-H3 (MiniMax, 2026; 33B) and Seedance 2.0/2.5 (ByteDance Seed, 2026a,b; closed-source). In the illustrated examples, these models offer higher visual quality, and Seedance 2.0 also produces better 2D portrait animation. Our method, fine-tuned from Wan2.2-5B (Wan et al., 2025), provides more precise camera control and uses a lighter backbone than H3. Detailed animation and reshooting comparisons are presented in Figures 12 and 13, respectively. Fine-tuning our framework from a stronger video backbone may improve visual quality while retaining precise control.

4. Comparison with Wan Animate beyond single view animation

We agree that our improvement over Wan-Animate (Cheng et al., 2025) on 2D animation metrics is modest. However, our ultimate goal is 4D animation, which requires consistency across generated views. To evaluate this, we generate multi-view animated videos with both methods, reconstruct a 4DGS from each method's outputs, and compare the renderings (Appendix D.2, Figure 10, p. 19). Our reconstructions exhibit fewer artifacts in these examples, indicating stronger multi-view consistency than Wan-Animate beyond the modest differences in 2D metrics.

Thanks for your response. Most of my concerns are addressed.

Reviewer_BJo2OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents UniPortrait4D, a unified framework for 4D portrait reshooting and animation using a video diffusion transformer. The method combines FLAME geometry and point-cloud rendering to improve multi-view consistency during reshooting, while introducing domain-adversarial expression encoding to reduce identity leakage during cross-identity animation. The generated multi-view videos can further support 4D Gaussian Splatting reconstruction for free-viewpoint rendering. Extensive quantitative and qualitative experiments demonstrate improvements in novel-view synthesis, multi-view consistency, identity preservation, and facial animation compared with existing baselines.

Critical strengths
  1. Simple yet effective unified framework. The paper presents a clear and well-motivated approach to unifying portrait reshooting and animation within a single video diffusion framework. The proposed components are straightforward and address specific challenges in the two tasks. In particular, combining point-cloud and FLAME conditioning improves multi-view consistency, while adversarial expression encoding supports identity-preserving animation. The overall methodology is clearly presented and demonstrates the effectiveness of a relatively simple design.

  2. Convincing experimental results and sufficient comparisons. The paper provides comprehensive qualitative and quantitative evaluations across both reshooting and animation tasks. The results consistently support the claimed improvements over existing baselines, while the ablation studies provide sufficient evidence for the effectiveness of the proposed components. Particularly, the generated novel views demonstrate visually consistent diffuse shading and specular appearance, supporting the overall 4D consistency. The downstream 4DGS reconstruction further validates the usefulness of the generated multi-view videos. Overall, the presented evidence is sufficient and convincing for the claimed contributions.

Critical weaknesses
  1. Limited discussion of illumination changes during reshooting. I do not identify any major weaknesses in the proposed method or experimental evaluation. One minor limitation is that the reshooting experiments primarily focus on viewpoint changes while preserving the original appearance. It remains unclear whether the framework could also support changes in illumination conditions, which would further extend its applicability to more general portrait reshooting scenarios.
Questions for authors

Does the proposed framework support illumination changes during reshooting? The presented results demonstrate promising consistency in diffuse and specular appearance across novel views. However, I am curious whether the framework can also support changes in the illumination environment, rather than preserving the lighting of the original video. Since illumination control is also relevant to practical reshooting applications, it would be interesting to understand whether such capability is supported by the current framework or would require additional modeling.

1 comment
AuthorsOpenReview ↗

Response to Reviewer BJo2

We thank the reviewer for the valuable comments. Our current framework preserves source illumination but does not support changing the lighting environment during reshooting. We have added this limitation and a possible extension to the main paper's Limitations section; further discussion is in Appendix D.6 (p. 22).

A possible extension would first apply a delighting model, then encode a target environment map or lighting reference video and inject the resulting features through cross-attention. Paired data from BecomingLit (Schmidt et al., 2025) and POLAR (Chen et al., 2026) could provide supervision, while joint multi-view training could improve consistency across viewpoints. We leave this extension to future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.