acceptodds
Under review as a conference paper at ICLR 2027

ViSA: 3D-Aware Video Shading for Real-Time Upper-Body Avatar Creation

Abstract

Generating high-fidelity upper-body 3D avatars from one-shot input image remains a significant challenge. Current 3D avatar generation methods, which rely on large reconstruction models, are fast and capable of producing stable body structures, but they often suffer from artifacts such as blurry textures and stiff, unnatural motion. In contrast, generative video models show promising performance by synthesizing photorealistic and dynamic results, but they frequently struggle with unstable behavior, including body structural errors and identity drift. To address these limitations, we propose a novel approach that combines the strengths of both paradigms. Our framework employs a 3D reconstruction model to provide robust structural and appearance priors, which in turn guides a real-time autoregressive video diffusion model for rendering. This process enables the model to synthesize high-frequency, photorealistic details and fluid dynamics in real time, effectively reducing texture blur and motion stiffness while preventing the structural inconsistencies common in video generation methods. By uniting the geometric stability of 3D reconstruction with the generative capabilities of video models, our method produces high-fidelity digital avatars with realistic appearance and dynamic, temporally coherent motion. Experiments demonstrate that our approach significantly reduces artifacts and achieves substantial improvements in visual quality over leading methods, providing a robust and efficient solution for real-time applications such as gaming and virtual reality.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Avatar Creation, Video Diffusion Model, 3D Reconstruction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_hQteOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents ViSA, a real-time video shading framework for upper-body avatar generation that combines 3D reconstruction with generative video synthesis. ViSA reconstructs a 3D-aware avatar representation from a single reference image and uses the rendered 3D-aware features to condition a distilled autoregressive video diffusion model. The framework further introduces an adversarial distribution-preservation objective to improve rendering sharpness during fine-tuning. Experiments evaluate the method in terms of visual quality, identity preservation, generation speed, and long-sequence synthesis.

Critical strengths

1.The paper addresses a practically significant problem: the balance between structural controllability and realism. ViSA combines an explicit 3D-aware avatar representation with a lightweight autoregressive video diffusion renderer to simultaneously ensure identity/geometric stability and high-frequency appearance details and motion realism. The overall design approach of the paper is sound.

2.The experimental results presented in this paper demonstrate a good balance between quality and efficiency. Compared to the 2D and 3D methods evaluated, ViSA achieved better results in terms of image quality and identity preservation while maintaining real-time generation speeds.

Critical weaknesses

1.The comparison with existing methods is relatively limited. The paper evaluates ViSA against only a small set of baselines, with limited coverage of more recent competing methods. Moreover, the implementation and evaluation protocols for methods from different paradigms are not described in sufficient detail, including their training/fine-tuning settings and inference configurations. This makes it difficult to fully assess the competitiveness of ViSA and the fairness of the quantitative comparisons reported in the table 1.

2.Some key design aspects still lack sufficient direct experimental validation. The advantages of 3D-aware feature conditioning and ADP are currently demonstrated primarily through qualitative results, while the individual contributions of the semantic features, low-level visual features, and general human prior in the reconstruction module are not analyzed. The effect of the two-stage progressive training strategy is also not directly evaluated. Consequently, although the full system achieves strong end-to-end performance, the specific contributions of several central design choices remain unclear.

Questions for authors

1.Could the authors provide quantitative results for the different conditioning inputs and for ADP, rather than only qualitative comparisons? This would help clarify how much each design contributes to visual quality, identity preservation, and temporal consistency.

2.Could the authors further clarify the contribution of the three reconstruction features—semantic features, low-level visual features, and the general human prior—and the role of the two-stage progressive training strategy?

3.Could the authors clarify the evaluation protocol for the compared methods, including whether official checkpoints are used, whether retraining or fine-tuning is performed, and whether the same reference images, driving motions, resolutions, and inference settings are used across methods?

4.Could the authors further clarify the role of the pretrained WAN backbone in ADP, and whether the observed improvement mainly comes from adversarial supervision itself or from using the pretrained video backbone as the discriminator prior?

3 comments
AuthorsOpenReview ↗

Official Comment by Authors

We appreciate the recognition of the framework's motivation and its balance between structural control, visual quality, and efficiency.

[W1.1] The comparison with existing methods is limited.

Thank you for your comment. We argue that our baseline selection covers the two main methodological paradigms relevant to our task, with each testing a different aspect of ViSA's capabilities.

Video-based methods (Champ, MimicMotion, and VACE) exploit generative priors to synthesize detailed appearance and expressive motion, but their multi-step generation incurs substantial computational cost. Comparisons with these methods demonstrate that ViSA achieves better image quality and identity preservation while substantially improving generation speed, as reported in Table 1.

Gaussian-based methods, represented by GUAVA, offer explicit geometric control, stable identity, and highly efficient rendering. This comparison tests whether our generative shader improves upon direct Gaussian rendering. The results show that ViSA achieves better image quality and identity preservation while retaining fast streaming generation at 15.15 FPS, although GUAVA remains faster.

Together, these comparisons assess ViSA against both generative video models and an efficient 3D-avatar system. Their coverage therefore extends across complementary strengths and directly supports our central claim: ViSA improves visual quality and identity preservation while offering a favorable quality–speed trade-off. We will make this rationale clearer in the revised paper.

[W1.2] Implementation and evaluation protocols are insufficiently described.

Thank you for your comment. We summarize the implementation details and evaluation protocols below.

Implementation details. ViSA is initialized from the Self-Forcing 1.3B model and fine-tuned for 3,000 steps using Adam with a learning rate of 5×10−55\times10^{-5} and (β1,β2)=(0.9,0.999)(\beta_1,\beta_2)=(0.9,0.999). We adopt a progressive training strategy: the reconstruction module and video shader are first jointly optimized on short clips, after which the reconstruction module is frozen and the shader is trained on longer sequences. Within the video diffusion backbone, only the input layer for the 3D-aware features and the self-attention blocks are updated. At inference, ViSA generates videos at 480p resolution using four denoising steps, achieving 15.15 FPS on a single NVIDIA A100 GPU.

Dataset and test split. We curate 46,000 clips from the public SpeakerVid-5M dataset. The processing pipeline includes motion and visibility filtering, SMPL-X/FLAME parameter estimation and refinement, and reprojection-error filtering, as described in Appendix A. Our test set contains 100 randomly selected subject IDs, each contributing one video clip. These test clips are excluded from training.

Evaluation protocols. We evaluate both self-reenactment and cross-reenactment. For self-reenactment, the first frame of each test video serves as the reference image, and the motion sequence from the same video provides the driving signals. We compute PSNR, SSIM, and LPIPS between generated frames and their corresponding ground-truth frames, and assess facial identity preservation using ArcFace-based IPS-self. For cross-reenactment, a source identity is animated using motion from a different subject. Since paired ground-truth videos are unavailable, we evaluate identity preservation using IPS-cross. Following the distance function in LHM's evaluation implementation, IPS-cross measures the ℓ1\ell_1 distance between ArcFace features extracted from the face regions of the reference image and generated frames; lower values indicate better identity preservation.

Background handling. The submitted image-quality metrics for ViSA include the background. Since GUAVA does not generate backgrounds, its metrics are computed on the human foreground, following its original evaluation protocol. To provide a comparison under matched foreground evaluation, we additionally report foreground-only results for ViSA alongside GUAVA.

We will consolidate these details in the revised paper, correct the description of IPS-cross, and provide a clearer account of the baseline implementations and inference configurations.

AuthorsOpenReview ↗

Official Comment by Authors

[W2 & Q1] Key design aspects lack direct validation, including 3D-aware conditioning, ADP, reconstruction features, and progressive training.

Thank you for the suggestion. Table below reports quantitative comparisons for the conditioning inputs, ADP, and progressive training on the same held-out test set. We also clarify how the three reconstruction features contribute to the conditioning representation.

Variant                       PSNR ↑  SSIM ↑  LPIPS ↓  IPS-self ↓  IPS-cross ↓
Joint conditioning             20.59    0.81    0.047       0.041        0.065
SMPL-X conditioning            21.83    0.83    0.045       0.042        0.066
Without reference KV cache     17.66    0.80    0.098       0.052        0.074
Without ADP                    21.42    0.82    0.046       0.040        0.064
Without progressive training   21.65    0.85    0.044       0.038        0.061
Full ViSA (Ours)               22.09    0.87    0.043       0.037        0.060

3D-aware conditioning. Rendered 3D features outperform both joint and SMPL-X conditioning on all five metrics. PSNR reaches 22.09 dB with rendered features, compared with 20.59 dB for joint conditioning and 21.83 dB for SMPL-X conditioning. IPS-cross decreases from 0.065 and 0.066, respectively, to 0.060, indicating better preservation of the source identity when the driving motion comes from a different subject.

Adversarial distribution preservation (ADP). With 3D-feature conditioning held fixed, adding ADP improves PSNR from 21.42 to 22.09 dB and SSIM from 0.82 to 0.87. LPIPS decreases from 0.046 to 0.043, consistent with the reduction in blur and artifacts shown in Fig. 6. Both identity distances also decrease.

Reconstruction features. The reconstruction module combines three sources of information. DINOv2 features encode subject-specific semantics, intermediate WAN-VAE features capture local appearance details, and a learned embedding at each canonical SMPL-X vertex supplies a human prior shared across subjects. The shader receives their fused representation rendered in the target pose. The conditioning ablation evaluates this representation as a whole. Isolating the contribution of each feature source would require removing them one at a time.

Progressive training. We first train the reconstruction module and shader jointly on short clips to align the rendered features with the shader. We then freeze the reconstruction module and train the shader on longer sequences. Removing this strategy lowers PSNR from 22.09 to 21.65 dB and SSIM from 0.87 to 0.85. LPIPS, IPS-self, and IPS-cross each increase by 0.001. The two-stage strategy thus gives consistent gains across the reported metrics, with the clearest improvements in PSNR and SSIM. We will include these ablations in the revised paper.

[Q2] Could the authors clarify the contributions of the three reconstruction features and the role of the two-stage progressive training strategy?

Reconstruction features. As described in Sec. 3.2, the three feature types serve complementary roles:

  • Semantic features: DINOv2 provides subject-dependent semantic information to guide reconstruction.

  • Low-level visual features: intermediate WAN-VAE features capture fine-grained local appearance details that complement the semantic features.

  • General human prior: a learned per-vertex embedding table provides subject-independent prior information associated with the canonical SMPL-X template.

Together, these features combine subject-specific semantics and appearance with a shared human prior, providing a richer representation for reconstruction and supporting higher-quality avatar synthesis.

Two-stage progressive training. The stages address different optimization needs. Stage 1 jointly trains the reconstructor and shader on short clips, aligning the learned 3D features with the pretrained video model. Stage 2 freezes the reconstructor and trains the shader on longer clips, exposing it to longer generated histories while keeping the conditioning representation fixed (Sec. 3.5 and Appendix D).

[Q4] What is the role of the pretrained WAN backbone in ADP, and does the improvement come from adversarial supervision or the discriminator prior?

Thank you for your question. The improvement mainly comes from the adversarial supervision. WAN pretrained backbone is intended to make this signal more stable by initialing the discriminator with large pretrained models to extract meaningful features.

My main concern about the reconstruction features remains. I was asking for controlled ablations that remove each of the three feature sources individually to measure its contribution. The current comparison evaluates the fused representation as a whole. The ADP ablation also leaves open how much of the gain comes from the pretrained WAN discriminator. Given these remaining concerns, I will maintain my original rating of 2

Reviewer_MdykOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents ViSA, a method for real-time upper-body human animation from a single identity image, driven by body pose and facial expression. It first uses a feed-forward reconstruction model to obtain learnable features, which are animated and rendered into 2D to serve as the condition for the generative module. The second part is a distilled few-step autoregressive video model, in which the identity image is injected at the first latent frame. An adversarial distribution preservation loss is introduced to improve the quality of the final rendered videos. Experimental results show the best image metrics and identity scores among the compared methods.

Critical strengths
  1. The algorithm design perserves real-time speed while does not lose the rendering performance.

  2. The results are strong. ViSA clearly outperforms baselines and the long video inference looks stable over time.

Critical weaknesses
  1. FlashMotion (CVPR 2026) and AAPT (NeurIPS 2025) both use similar adversarial losses to improve the quality of few-step video generation. The paper neither compares ADP with these methods nor discusses in depth how it differs from them, which reduces the novelty of ADP.

  2. GAS (ICCV 2025) and ELITE (CVPR 2026) already explore a similar reconstruction-guided generation idea, using renderings of a reconstructed avatar to guide a diffusion model. However, the paper does not discuss either work.

  3. The quality ablations are qualitative only. Fig. 6 compares the conditioning inputs and ADP on selected examples, and Table 2 quantifies only latency. Without quantitative results on the test set, the contribution of each component to quality, identity, and motion fidelity cannot be assessed.

  4. The third listed contribution restates experimental results rather than a contribution.

Questions for authors
  1. What does ADP add over FlashMotion and AAPT? A controlled comparison with their discriminator designs would clarify this.

  2. The overall design is sound and the results are promising, but the quality ablations are only qualitative. Could you report quantitative ablations for the conditioning inputs, ADP? Without quantitative ablation results, I would consider lowering my score. Quantitative results for the timestep schedule and pose/expression accuracy against the driving signals would further strengthen the paper.

  3. Reporting a drift metric over time, e.g., IPS-cross computed per 10-second segment of minute-long videos, would strengthen the paper.

  4. Will the curated dataset, annotations, test split, code, and weights be released?

3 comments
AuthorsOpenReview ↗

Official Comment by Authors

We thank the reviewer for recognizing the quality–efficiency balance and the promising long-sequence results.

[W1 & Q1] FlashMotion and AAPT use similar adversarial losses, but their differences from ADP are not discussed in depth.

Thank you for your question. We agree that these methods share the use of adversarial supervision to improve generation quality. FlashMotion uses diffusion and adversarial losses to align a trajectory adapter with a distilled generator, while AAPT uses consistency-distilled initialization followed by adversarial training for one-step autoregressive generation.

Our emphasis is on preserving pretrained generative priors while introducing 3D-feature conditioning into an already-distilled model. Directly fine-tuning such a distilled model with paired reconstruction losses alone can degrade its learned visual realism. Therefore, we introduce ADP to provide an additional distribution-level supervision signal that regularizes this adaptation, encourages realistic outputs, and mitigates quality degradation.

With ADP, our model can incorporate the new 3D conditioning while better preserving visual realism and fine appearance details, enabling high-quality avatar synthesis with explicit motion control. We will revise the paper to emphasize this motivation, discuss the difference with FlashMotion and AAPT, and the specific role of ADP in our adaptation setting.

[W2] GAS and ELITE already explore a similar reconstruction-guided generation idea, but neither work is discussed.

Thank you for pointing out these relevant works. GAS combines single-image 3D reconstruction with video diffusion, conditioning on encoded NeRF RGB renderings and SMPL normal maps. It trains the reconstruction and diffusion modules separately, keeping the NeRF fixed during diffusion training, and generates videos using a clip-based diffusion model. ViSA learns a direct feature interface between reconstruction and streaming generation. Our reconstruction module combines semantic features, low-level visual features, and a learned human prior to construct an animatable Gaussian representation. This representation renders pose-aligned features that directly condition a four-step autoregressive shader. During training, we jointly optimize the reconstruction module and shader so that the rendered features adapt to the generation task. Therefore, although both methods use 3D reconstruction to guide generation, GAS encodes rendered images for clip-based diffusion, whereas ViSA jointly learns the pose-aligned features that drive its streaming shader.

ELITE initializes a Gaussian head avatar from a monocular video and uses a rendering-guided, single-step diffusion enhancer for identity-specific adaptation at test time. ViSA animates the upper body from a single reference image using a shared autoregressive shader. This broader scope makes the animation task more challenging: ViSA must preserve the source identity while coordinating torso motion, hand gestures, and facial expressions over time.

We will discuss both works in the revised paper and clarify how ViSA differs in its conditioning inputs, training procedure, and animation scope.

AuthorsOpenReview ↗

Official Comment by Authors

[W3 & Q2] The quality ablations are qualitative only.

Thank you for the suggestion. We now report quantitative ablations for the conditioning inputs and ADP in Table 1. We evaluate all variants on the same held-out test set using the same protocol. The table also reports the effects of the reference KV cache and progressive training.

Variant                       PSNR ↑  SSIM ↑  LPIPS ↓  IPS-self ↓  IPS-cross ↓
Joint conditioning             20.59    0.81    0.047       0.041        0.065
SMPL-X conditioning            21.83    0.83    0.045       0.042        0.066
Without reference KV cache     17.66    0.80    0.098       0.052        0.074
Without ADP                    21.42    0.82    0.046       0.040        0.064
Without progressive training   21.65    0.85    0.044       0.038        0.061
Full ViSA (Ours)               22.09    0.87    0.043       0.037        0.060

Table 1. Quantitative ablations on the ViSA test set. The full-model results are from Table 1 of the submitted paper.

Conditioning inputs. Conditioning on rendered 3D features improves all five metrics over both joint and SMPL-X conditioning. For example, PSNR increases from 20.59 dB with joint conditioning and 21.83 dB with SMPL-X conditioning to 22.09 dB with our learned features. IPS-cross also decreases from 0.065 and 0.066, respectively, to 0.060, indicating better preservation of the reference identity when the driving motion comes from another subject. Figure 6 provides the complementary visual comparison between 2D keypoints, rendered 3D RGB, and rendered 3D features.

ADP. To isolate the effect of ADP, we keep the 3D-feature conditioning fixed and compare the full model with the same system trained without the adversarial term. Adding ADP improves PSNR from 21.42 to 22.09 dB, raises SSIM from 0.82 to 0.87, and reduces LPIPS from 0.046 to 0.043. Both identity distances also decrease. These results complement the sharper facial and clothing details observed in Fig. 6.

We will include these quantitative ablations in the revised paper.

[W4] The third listed contribution restates experimental results rather than a contribution.

Thank you for your suggestion. The third item was intended to highlight our experimental results. We will revise the contribution list to focus on our methodological contributions and present the experimental results separately.

[Q3] Could you report a drift metric over time, such as IPS-cross per 10-second segment of minute-long videos?

Thank you for your suggestion. We report IPS-cross for each consecutive 10-second segment of one-minute generated videos in the table below.

Time interval (s)   0–10  10–20  20–30  30–40  40–50  50–60
IPS-cross ↓        0.585  0.599  0.581  0.596  0.593  0.604

As shown in the table, IPS-cross remains relatively stable across the six segments, indicating consistent identity preservation with limited drift over one-minute generation. We will include these results in the revised paper.

[Q4] Will the curated dataset, annotations, test split, code, and weights be released?

Thank you for your question. Public release of the curated dataset, annotations, test split, code, and model weights is subject to internal approval. Therefore, we cannot make a firm commitment to releasing these resources at this stage. We recognize the importance of reproducibility and will explore the possibility of sharing selected resources where feasible and permitted.

Reviewer_MdykOpenReview ↗

Comments from Reviewer Mdyk

Thank you for the additional quantitative ablations and clarifications. The ADP ablation supports the usefulness of adversarial supervision in this task. However, the specific contribution of ADP relative to existing adversarial training approaches remains unclear. The relativistic objective and approximate R1/R2 regularization are established techniques, so please clarify which aspects of ADP are new and which are adapted from prior work.

If ADP’s contribution lies in its discriminator architecture or reference-conditioning design, please specify the changes relative to prior work and compare against an appropriate existing discriminator baseline. Controlled ablations should isolate the contribution of the newly introduced components while keeping the adversarial objective, regularization, and training setup unchanged. If ADP primarily applies established adversarial training to this task, its role and contribution should be described accordingly.

The gains over SMPL-X conditioning are modest in PSNR and LPIPS, although SSIM and identity-preservation metrics improve. The added quantitative table does not clearly identify the 3D-RGB conditioning baseline shown in Fig. 6 and Table 2. Please clarify whether the “SMPL-X conditioning” row corresponds to this setting. If not, please report the same quality and identity-preservation metrics for 3D-RGB conditioning under the same evaluation protocol.

Please also clarify the specific differences in the highlighted hand regions of Fig. 6(a) and explain why 3D-feature conditioning improves finger details over 3D-RGB conditioning. How well does the reconstructed Gaussian avatar represent individual fingers, and what additional cues do the learned features provide to the video shader beyond rendered RGB?

I understand that public release is subject to internal approval. If feasible, I would encourage the authors to share the curated dataset and training code to support reproducibility and future research.

Overall, while the results are strong, the current presentation and analysis leave me uncertain about the specific methodological contributions of ADP and the conditioning design relative to prior work. Clarification of these points would be important for my final assessment.

Reviewer_6qScOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

ViSA addresses avatar video generation, transferring source identity to target motion sequences. It employs a plethora of models, modules, and techniques, including SAM, DINO, SMPL, LBS, Gaussian Splatting, and adversarial training, and achieves good visual quality and identity preservation.

Critical strengths
  • Good quantitative and qualitative results.
  • Technical designs are well motivated
  • Paper well written and easy to follow
Critical weaknesses
  • Real-time claim is exaggerated: at 15 FPS, especially on H20 GPUs, against baseline 89.12.
  • In metrics computation, it's unclear if background color is taken into account/if a SAM crop out of the character is used for calculation. This is important for fair comparison as GUAVA's qualitative results do not seem to carry background.
  • Only one dataset is tested, and the dataset is curated by the authors, which might favour their designs.
  • Experiment and ablation sections are lean, but understandable given the detailed method section.
Questions for authors
  • Does metrics calculation compare the raw images vs ground truth, or the character cut out?
  • On what hardware are the methods timed for FPS?
2 comments
AuthorsOpenReview ↗

Official Comment by Authors

We appreciate the positive assessment of the results, technical motivation, and presentation.

[W1] Real-time claim is exaggerated.

Thank you for the helpful comment. We would like to clarify that our use of “real-time” is intended to convey that our method enables fast streaming generation, substantially faster than conventional multi-step video diffusion methods. Our method currently achieves approximately 15 FPS on a single A100 GPU, which falls within a similar throughput range to that reported by other “real-time” video generation methods, such as Self Forcing.

We agree that our current generation speed remains considerably lower than that of Gaussian-based methods. However, ViSA achieves better visual quality and identity preservation than GUAVA in our experiments, reflecting a quality–speed trade-off. We will revise the wording in the paper to explicitly state the achieved frame rate and clarify that a substantial speed gap remains relative to Gaussian-based methods such as GUAVA (89.12 FPS in our comparison).

[W2 & Q1] It is unclear whether background color is taken into account or a SAM character cutout is used for metric computation.

Thank you for the question. The metrics reported for ViSA are computed over the full images, including the backgrounds. For GUAVA, which does not generate backgrounds, we follow the evaluation protocol in its original paper and compute the metrics only over the human foreground. To provide a fair comparison and better assess the quality of the generated avatars, we additionally report foreground-only metrics for ViSA below, evaluating both methods only over the human region:

Method       PSNR ↑  SSIM ↑  LPIPS ↓
GUAVA         18.59    0.86    0.072
ViSA (ours)   23.89    0.91    0.039

ViSA also outperforms GUAVA under foreground-only evaluation, demonstrating that its advantage in avatar synthesis quality is maintained when the background is excluded. We will clarify the evaluation protocol and include these additional results in the revised paper.

AuthorsOpenReview ↗

Official Comment by Authors

[W3] Only one dataset is tested, and the dataset is curated by the authors.

Thank you for raising this concern. We would like to clarify that our 46,000 clips are curated from the publicly available SpeakerVid-5M dataset, rather than collected specifically for ViSA. As described in Appendix A, the preprocessing includes motion/visibility filtering and SMPL-X/FLAME fitting followed by reprojection-error filtering. These steps provide reliable control annotations for training and evaluation.

To our knowledge, there is no widely adopted benchmark with a standardized evaluation protocol for this setting, and existing studies typically use their own curated test sets. Some evaluations involve relatively few source identities; for example, GUAVA evaluates cross-reenactment using 10 source images driven by 8 videos.

Our evaluation uses 100 randomly sampled test cases that are kept separate from the training data, and the compared methods are evaluated on the same test set. Random sampling and train–test separation reduce the risk of selective evaluation and provide a systematic basis for comparing methods. We believe these results support our empirical conclusions within the evaluated setting. We will clarify the dataset construction, test sampling, and train–test separation in the revised paper.

[W4] Experiment and ablation sections are lean.

Thank you for the suggestion. To complement the qualitative comparisons in Fig. 6, we provide quantitative ablations in the following Table. We evaluate the conditioning input, ADP, the reference KV cache, and progressive training on the same test set using the same image-quality and identity-preservation metrics.

Variant                       PSNR ↑  SSIM ↑  LPIPS ↓  IPS-self ↓  IPS-cross ↓
Joint conditioning             20.59    0.81    0.047       0.041        0.065
SMPL-X conditioning            21.83    0.83    0.045       0.042        0.066
Without reference KV cache     17.66    0.80    0.098       0.052        0.074
Without ADP                    21.42    0.82    0.046       0.040        0.064
Without progressive training   21.65    0.85    0.044       0.038        0.061
Full ViSA (Ours)               22.09    0.87    0.043       0.037        0.060

Conditioning inputs. Figure 6 compares 2D keypoints, rendered 3D RGB, and rendered 3D features qualitatively. The quantitative results show that conditioning on learned 3D features achieves better scores on all five metrics than either joint or SMPL-X conditioning. For example, PSNR increases from 20.59 and 21.83 dB, respectively, to 22.09 dB, while IPS-cross decreases from 0.065 and 0.066 to 0.060.

ADP. With the same 3D-feature conditioning, adding the adversarial term improves PSNR from 21.42 to 22.09 dB and SSIM from 0.82 to 0.87. LPIPS also decreases from 0.046 to 0.043, consistent with the sharper appearance observed in Fig. 6.

Reference conditioning and progressive training. Removing the reference KV cache produces the largest degradation among the tested variants. LPIPS increases from 0.043 to 0.098 and IPS-cross from 0.060 to 0.074, supporting the role of persistent access to the reference image in preserving appearance and identity. Progressive training yields smaller but consistent gains across all five metrics. We will include these quantitative results and their interpretation in the revised experimental section.

[Q2] On what hardware are the methods timed for FPS?

Thanks for the question. The reported inference throughput of 15.15 FPS is measured on a single NVIDIA A100, as stated in Sec. 4.2 and Table 2.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to all reviewers

Thank you for the thoughtful reviews. The reviews ask how much the 3D conditioning and ADP contribute and how to interpret the quality and speed comparisons. The revised manuscript marks changes in blue. Three additions address the main questions.

  • Quantitative ablations (Appendix H, Table 5). We compare conditioning inputs, adversarial distribution preservation (ADP), persistent reference conditioning, and progressive training on the same held-out set. With 3D-feature conditioning fixed, ADP raises SSIM from 0.82 to 0.87. Removing access to the cached reference increases IPS-cross from 0.060 to 0.074.
  • Relation to prior work (Sec. 2, Appendices I–J). We explain how ViSA learns the features passed from reconstruction to the streaming shader, and why an already-distilled shader benefits from adversarial supervision when it learns this new conditioning input. Appendix J discusses GAS, ELITE, FlashMotion, and AAPT in detail.
  • Evaluation and speed (Sec. 4.2, Appendices G–H). At 480p, ViSA streams at 15.15 FPS on one A100. Under the same foreground-only evaluation, ViSA achieves higher PSNR (23.89 vs. 18.59 dB) and lower LPIPS (0.039 vs. 0.072) than GUAVA (Table 4).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.