- Rating
- 3(+1)
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper presents UniPortrait4D, a unified video diffusion transformer for portrait reshooting and cross-identity animation. For reshooting, the method combines point-cloud renderings and FLAME geometry with view attention to improve multi-view consistency. For animation, it employs a high-dimensional expression encoder with gradient-reversal-based adversarial training to suppress driving-identity information while preserving facial expressions. The generated multi-view videos are further used for 4D Gaussian Splatting reconstruction, enabling free-viewpoint rendering. Experiments on AVA-256 and FEED demonstrate improvements over existing reshooting and animation baselines in terms of generation fidelity, multi-view consistency, identity preservation, and motion transfer. Ablation studies further evaluate the effectiveness of the proposed components and unified training.
Critical strengths
1. Well-motivated problem formulation and unified framework. The paper identifies a meaningful research gap by recognizing the complementary characteristics of portrait reshooting and animation. While reshooting benefits from multi-view supervision but suffers from limited identity diversity, animation can leverage diverse identities and facial motions from monocular videos. The proposed unified framework is a well-motivated approach to exploiting these complementary sources of supervision. Furthermore, the hybrid geometry conditioning and identity-adversarial expression encoding are thoughtfully designed to address the respective challenges of the two tasks.
2. Strong empirical performance and comprehensive evaluation. The proposed method demonstrates strong performance compared to existing baselines on both portrait reshooting and animation tasks. In particular, the evaluation considers not only the fidelity of generated videos but also downstream 4DGS reconstruction quality and multi-view consistency, which are well aligned with the paper's objectives. Additionally, the ablation studies provide empirical evidence supporting the effectiveness of the proposed hybrid geometry conditioning and identity-adversarial training.
Critical weaknesses
-
Insufficient details on unified training. Although unified training is a central contribution, the paper does not clearly specify the training objective, task sampling strategy, or how the reshooting and animation tasks are jointly optimized. These missing details make the proposed framework difficult to fully understand and reproduce.
-
Insufficient validation of unified training. Table 3(b) only demonstrates that reshooting supervision benefits animation. The reverse direction, whether animation training improves reshooting performance, is not evaluated, leaving the claimed mutual benefits insufficiently supported.
-
Lack of clarity in methodology and presentation. Several notations (e.g., and ) are introduced without clear definitions. Moreover, Figure 3 lacks sufficient explanation in the main text, making it difficult to understand the roles of its components and the overall training pipeline.
Questions for authors
-
Clarification of unified training. Could the authors provide more details on the unified training procedure, including the training objective, task sampling strategy, and how the two tasks are jointly optimized? Additionally, is the pretrained expression encoder frozen or jointly fine-tuned during unified training?
-
Ablation on unified training. Table 3(b) demonstrates that reshooting supervision benefits animation. However, the reverse direction is not evaluated. Could the authors provide a comparison between reshooting-only and unified training to verify whether animation supervision also improves reshooting performance?
-
Evaluation of multi-view animation. While Table 3(b) provides some evidence of improved multi-view consistency, the evaluation of 4D animation remains limited. Could the authors provide quantitative comparisons with relevant prior methods, such as MVP4D, GeoDiff4D, and FactorPortrait, where applicable? In particular, comparisons of multi-view consistency and downstream 4D avatar reconstruction quality would help substantiate the claimed 4D animation capability.
4 comments
Official comment by reviewer uEt1
Thank you for the response and additional qualitative results. While the explanation and Figure 9(a) are helpful, I believe my concern regarding the mutual benefits of unified training remains insufficiently addressed.
My original question specifically requested a controlled ablation comparing reshooting-only training and unified training to verify whether animation supervision improves reshooting performance. Although the additional qualitative results suggest potential improvements, they do not clearly isolate the contribution of animation training.
I would appreciate a quantitative comparison under comparable training settings to substantiate this claim. Otherwise, I believe the claim regarding the bidirectional benefits of unified training should be stated more cautiously.
Response to Reviewer uEt1
Thank you for the suggestion. We provide below a quantitative comparison between reshooting-only and unified training. We evaluate on 50 test identities from AVA-256, none of which appear in the training data. We use the same evaluation metrics as Table 1 in the main paper's Comparison section: PSNR, SSIM, and LPIPS for Generated-to-GT, 4DGS-to-GT, and generated-view consistency (GVC).
Both models use the same architecture. The reshooting-only model is trained for 40,000 iterations using only NeRSemble (3D data). The unified model is trained for 40,000 iterations (20,000 animation iterations and 20,000 reshooting iterations) using VFHQ (2D data) and NeRSemble (3D data). We find that the unified model outperforms the reshooting-only model under comparable training settings, providing evidence that animation supervision improves reshooting performance.
| Training | Gen-to-GT | 4DGS-to-GT | GVC | ||||||
|---|---|---|---|---|---|---|---|---|---|
| PSNR ↑ | SSIM ↑ | LPIPS ↓ | PSNR ↑ | SSIM ↑ | LPIPS ↓ | PSNR ↑ | SSIM ↑ | LPIPS ↓ | |
| Reshooting-only | 19.51 | 0.7332 | 0.2841 | 19.13 | 0.7051 | 0.3171 | 23.69 | 0.7658 | 0.2453 |
| Unified | 21.40 | 0.7740 | 0.2664 | 20.76 | 0.7589 | 0.2796 | 24.33 | 0.7872 | 0.1970 |
Unified training improves all three groups of metrics. We believe that the additional animation data, covering more diverse expressions and lighting conditions, improve reshooting generalization to in-the-wild inputs, yielding clearer facial details, as illustrated in Figure 9(a) (p. 19). In Figure 9(a), “w/o Animation” denotes the reshooting-only results, while “Ours” denotes the results of unified training. We have also added this quantitative comparison to Appendix D.1 as Table 4 (p. 20).
Official comment by reviewer uEt1
Thank you for your detailed response. While this topic is somewhat outside my primary area of expertise (as reflected in my confidence rating), I found your clarifications helpful and believe they adequately address my concerns. I have therefore decided to raise my score.
Response to Reviewer uEt1
We thank the reviewer for the valuable feedback on the training procedure, task interaction, notation, and 4D animation evaluation. Additional experiments are in Appendix D, and the notation and training-sampling strategy have been clarified in the revised manuscript.
1. Unified training objective and sampling
Training consists of three stages. First, we train the pose/expression encoding and appearance reconstruction framework through DeX-Portrait-style self-supervised reconstruction (Shi et al., 2026). Second, we freeze the expression encoder and train an identity probe, then jointly optimize the encoder, probe, and reconstruction pathway through gradient reversal while retaining the reconstruction objective. Third, we freeze the resulting expression encoder and train the shared DiT with the standard flow-matching objective.
During DiT training, reshooting, monocular animation, and multi-view animation are sampled at a 5:2:3 ratio. Reshooting uses synchronized NeRSemble videos from different cameras, with one view as the source and target-camera videos as ground truth. Monocular animation uses a same-identity reference image and driving clip, supervised by the driving video. Multi-view animation uses a same-identity reference and synchronized clips; the driving camera may differ from the target cameras, and one to four target views are sampled. All modes use flow matching against their ground-truth target videos. The stage-wise optimization, sampling, and supervision are documented in Appendix B.1, Three-Stage Training and Task Sampling (pp. 15–16).
2. Does animation training also benefit reshooting
Yes. Figure 9(a) (Appendix D.1, p. 18) shows improved reshooting on challenging in-the-wild videos, including exaggerated expressions, tongue/eye/mouth details, and source illumination. At the data level, animation introduces abundant monocular videos with more varied expressions and lighting. At the architecture level, source-video expression latents are also injected through cross-attention during reshooting, providing additional guidance for complex expressions. Together, these provide a route for animation training to improve reshooting generalization.
3. Notation and Figure 3
Id denotes the driving image supplied to the expression encoder Fϕ. zt,v denotes the noisy video latent at video time index t and target-view index v. We have added these definitions to Sections 3.3 and 3.2, respectively. We have also expanded the self-supervised initialization paragraph in Section 3.3 to explain the appearance encoder, pose-based feature warping, expression encoder, and reconstruction decoder, with an explicit reference to Figure 3; architectural details are provided in Appendix B. The added notation definitions and initialization explanation are highlighted in blue on pp. 4–5. These revisions distinguish motion-encoder initialization from GRL post-training and the subsequent DiT training.
4. Comparisons with 4D animation methods
We could not obtain publicly released implementations of GeoDiff4D (Xu et al., 2026) and FactorPortrait (Tang et al., 2025) for this revision. We therefore compare with MVP4D (Taubner et al., 2025a), UIKA (Wu et al., 2026), and CAP4D (Taubner et al., 2025b). The added evaluation uses 50 identities selected from AVA-256 and reports cross-identity reenactment only. Qualitative results compare reconstructed 4DGS avatar renderings in Figure 14; quantitative results are given in Table 4 (Appendix D.5, p. 21):
Our method achieves the highest CSIM and lowest AED, APD, and MET3R among these baselines. MET3R (Asim et al., 2025) is measured between synchronized frames from the generated multi-view videos, not from the reconstructed avatar renderings; lower is better. View attention supports cross-view information exchange, improving consistency and reducing reconstruction artifacts, while GRL reduces driver-identity leakage and improves reference-identity preservation. The qualitative examples and quantitative results support these advantages.