- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents ViSA, a real-time video shading framework for upper-body avatar generation that combines 3D reconstruction with generative video synthesis. ViSA reconstructs a 3D-aware avatar representation from a single reference image and uses the rendered 3D-aware features to condition a distilled autoregressive video diffusion model. The framework further introduces an adversarial distribution-preservation objective to improve rendering sharpness during fine-tuning. Experiments evaluate the method in terms of visual quality, identity preservation, generation speed, and long-sequence synthesis.
Critical strengths
1.The paper addresses a practically significant problem: the balance between structural controllability and realism. ViSA combines an explicit 3D-aware avatar representation with a lightweight autoregressive video diffusion renderer to simultaneously ensure identity/geometric stability and high-frequency appearance details and motion realism. The overall design approach of the paper is sound.
2.The experimental results presented in this paper demonstrate a good balance between quality and efficiency. Compared to the 2D and 3D methods evaluated, ViSA achieved better results in terms of image quality and identity preservation while maintaining real-time generation speeds.
Critical weaknesses
1.The comparison with existing methods is relatively limited. The paper evaluates ViSA against only a small set of baselines, with limited coverage of more recent competing methods. Moreover, the implementation and evaluation protocols for methods from different paradigms are not described in sufficient detail, including their training/fine-tuning settings and inference configurations. This makes it difficult to fully assess the competitiveness of ViSA and the fairness of the quantitative comparisons reported in the table 1.
2.Some key design aspects still lack sufficient direct experimental validation. The advantages of 3D-aware feature conditioning and ADP are currently demonstrated primarily through qualitative results, while the individual contributions of the semantic features, low-level visual features, and general human prior in the reconstruction module are not analyzed. The effect of the two-stage progressive training strategy is also not directly evaluated. Consequently, although the full system achieves strong end-to-end performance, the specific contributions of several central design choices remain unclear.
Questions for authors
1.Could the authors provide quantitative results for the different conditioning inputs and for ADP, rather than only qualitative comparisons? This would help clarify how much each design contributes to visual quality, identity preservation, and temporal consistency.
2.Could the authors further clarify the contribution of the three reconstruction features—semantic features, low-level visual features, and the general human prior—and the role of the two-stage progressive training strategy?
3.Could the authors clarify the evaluation protocol for the compared methods, including whether official checkpoints are used, whether retraining or fine-tuning is performed, and whether the same reference images, driving motions, resolutions, and inference settings are used across methods?
4.Could the authors further clarify the role of the pretrained WAN backbone in ADP, and whether the observed improvement mainly comes from adversarial supervision itself or from using the pretrained video backbone as the discriminator prior?
3 comments
Official Comment by Authors
[W2 & Q1] Key design aspects lack direct validation, including 3D-aware conditioning, ADP, reconstruction features, and progressive training.
Thank you for the suggestion. Table below reports quantitative comparisons for the conditioning inputs, ADP, and progressive training on the same held-out test set. We also clarify how the three reconstruction features contribute to the conditioning representation.
Variant PSNR ↑ SSIM ↑ LPIPS ↓ IPS-self ↓ IPS-cross ↓
Joint conditioning 20.59 0.81 0.047 0.041 0.065
SMPL-X conditioning 21.83 0.83 0.045 0.042 0.066
Without reference KV cache 17.66 0.80 0.098 0.052 0.074
Without ADP 21.42 0.82 0.046 0.040 0.064
Without progressive training 21.65 0.85 0.044 0.038 0.061
Full ViSA (Ours) 22.09 0.87 0.043 0.037 0.060
3D-aware conditioning. Rendered 3D features outperform both joint and SMPL-X conditioning on all five metrics. PSNR reaches 22.09 dB with rendered features, compared with 20.59 dB for joint conditioning and 21.83 dB for SMPL-X conditioning. IPS-cross decreases from 0.065 and 0.066, respectively, to 0.060, indicating better preservation of the source identity when the driving motion comes from a different subject.
Adversarial distribution preservation (ADP). With 3D-feature conditioning held fixed, adding ADP improves PSNR from 21.42 to 22.09 dB and SSIM from 0.82 to 0.87. LPIPS decreases from 0.046 to 0.043, consistent with the reduction in blur and artifacts shown in Fig. 6. Both identity distances also decrease.
Reconstruction features. The reconstruction module combines three sources of information. DINOv2 features encode subject-specific semantics, intermediate WAN-VAE features capture local appearance details, and a learned embedding at each canonical SMPL-X vertex supplies a human prior shared across subjects. The shader receives their fused representation rendered in the target pose. The conditioning ablation evaluates this representation as a whole. Isolating the contribution of each feature source would require removing them one at a time.
Progressive training. We first train the reconstruction module and shader jointly on short clips to align the rendered features with the shader. We then freeze the reconstruction module and train the shader on longer sequences. Removing this strategy lowers PSNR from 22.09 to 21.65 dB and SSIM from 0.87 to 0.85. LPIPS, IPS-self, and IPS-cross each increase by 0.001. The two-stage strategy thus gives consistent gains across the reported metrics, with the clearest improvements in PSNR and SSIM. We will include these ablations in the revised paper.
[Q2] Could the authors clarify the contributions of the three reconstruction features and the role of the two-stage progressive training strategy?
Reconstruction features. As described in Sec. 3.2, the three feature types serve complementary roles:
-
Semantic features: DINOv2 provides subject-dependent semantic information to guide reconstruction.
-
Low-level visual features: intermediate WAN-VAE features capture fine-grained local appearance details that complement the semantic features.
-
General human prior: a learned per-vertex embedding table provides subject-independent prior information associated with the canonical SMPL-X template.
Together, these features combine subject-specific semantics and appearance with a shared human prior, providing a richer representation for reconstruction and supporting higher-quality avatar synthesis.
Two-stage progressive training. The stages address different optimization needs. Stage 1 jointly trains the reconstructor and shader on short clips, aligning the learned 3D features with the pretrained video model. Stage 2 freezes the reconstructor and trains the shader on longer clips, exposing it to longer generated histories while keeping the conditioning representation fixed (Sec. 3.5 and Appendix D).
[Q4] What is the role of the pretrained WAN backbone in ADP, and does the improvement come from adversarial supervision or the discriminator prior?
Thank you for your question. The improvement mainly comes from the adversarial supervision. WAN pretrained backbone is intended to make this signal more stable by initialing the discriminator with large pretrained models to extract meaningful features.
My main concern about the reconstruction features remains. I was asking for controlled ablations that remove each of the three feature sources individually to measure its contribution. The current comparison evaluates the fused representation as a whole. The ADP ablation also leaves open how much of the gain comes from the pretrained WAN discriminator. Given these remaining concerns, I will maintain my original rating of 2
Official Comment by Authors
We appreciate the recognition of the framework's motivation and its balance between structural control, visual quality, and efficiency.
[W1.1] The comparison with existing methods is limited.
Thank you for your comment. We argue that our baseline selection covers the two main methodological paradigms relevant to our task, with each testing a different aspect of ViSA's capabilities.
Video-based methods (Champ, MimicMotion, and VACE) exploit generative priors to synthesize detailed appearance and expressive motion, but their multi-step generation incurs substantial computational cost. Comparisons with these methods demonstrate that ViSA achieves better image quality and identity preservation while substantially improving generation speed, as reported in Table 1.
Gaussian-based methods, represented by GUAVA, offer explicit geometric control, stable identity, and highly efficient rendering. This comparison tests whether our generative shader improves upon direct Gaussian rendering. The results show that ViSA achieves better image quality and identity preservation while retaining fast streaming generation at 15.15 FPS, although GUAVA remains faster.
Together, these comparisons assess ViSA against both generative video models and an efficient 3D-avatar system. Their coverage therefore extends across complementary strengths and directly supports our central claim: ViSA improves visual quality and identity preservation while offering a favorable quality–speed trade-off. We will make this rationale clearer in the revised paper.
[W1.2] Implementation and evaluation protocols are insufficiently described.
Thank you for your comment. We summarize the implementation details and evaluation protocols below.
Implementation details. ViSA is initialized from the Self-Forcing 1.3B model and fine-tuned for 3,000 steps using Adam with a learning rate of 5×10−5 and (β1,β2)=(0.9,0.999). We adopt a progressive training strategy: the reconstruction module and video shader are first jointly optimized on short clips, after which the reconstruction module is frozen and the shader is trained on longer sequences. Within the video diffusion backbone, only the input layer for the 3D-aware features and the self-attention blocks are updated. At inference, ViSA generates videos at 480p resolution using four denoising steps, achieving 15.15 FPS on a single NVIDIA A100 GPU.
Dataset and test split. We curate 46,000 clips from the public SpeakerVid-5M dataset. The processing pipeline includes motion and visibility filtering, SMPL-X/FLAME parameter estimation and refinement, and reprojection-error filtering, as described in Appendix A. Our test set contains 100 randomly selected subject IDs, each contributing one video clip. These test clips are excluded from training.
Evaluation protocols. We evaluate both self-reenactment and cross-reenactment. For self-reenactment, the first frame of each test video serves as the reference image, and the motion sequence from the same video provides the driving signals. We compute PSNR, SSIM, and LPIPS between generated frames and their corresponding ground-truth frames, and assess facial identity preservation using ArcFace-based IPS-self. For cross-reenactment, a source identity is animated using motion from a different subject. Since paired ground-truth videos are unavailable, we evaluate identity preservation using IPS-cross. Following the distance function in LHM's evaluation implementation, IPS-cross measures the ℓ1 distance between ArcFace features extracted from the face regions of the reference image and generated frames; lower values indicate better identity preservation.
Background handling. The submitted image-quality metrics for ViSA include the background. Since GUAVA does not generate backgrounds, its metrics are computed on the human foreground, following its original evaluation protocol. To provide a comparison under matched foreground evaluation, we additionally report foreground-only results for ViSA alongside GUAVA.
We will consolidate these details in the revised paper, correct the description of IPS-cross, and provide a clearer account of the baseline implementations and inference configurations.