acceptodds
Under review as a conference paper at ICLR 2027

TaoHeadVerse: Learning Expressive Full-Head Avatars Beyond Representation Confounding

12 views

Abstract

Personalized full-head avatars for mobile XR and telepresence require faithful animation and efficient rendering. Creating such avatars from one or a few portraits remains challenging because it requires completing unseen regions, preserving source identity under cross-identity driving, and modeling person-specific expression responses. We present TaoHeadVerse, a feed-forward framework that jointly estimates motion and predicts an explicit mesh-bound Gaussian avatar. To address representation confounding in joint avatar-motion learning and mixed-source training, Neutral Identity Anchoring limits identity leakage into expression components, while Adversarial Domain Alignment discourages canonical features from encoding dataset-level acquisition attributes. The predicted asset stores identity-adapted mesh blendshapes and Gaussian appearance bases, enabling animation through basis mixing without per-frame neural avatar decoding. Experiments on VFHQ, AVA256, and RenderMe-360 show improved reconstruction, identity preservation, and full-head completion with competitive expression fidelity. Given precomputed driving controls, four concurrent avatars achieve animation and rendering throughput above 60 FPS on a shared canvas on both mobile devices.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Animatable Head Avatars, 3D Gaussian Splatting, Few-Shot 3D Reconstruction, Cross-Identity Expression Transfer, Full-Head Reconstruction, Real-Time Mobile Animation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_og21OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents TaoHeadVerse, a feed-forward framework that reconstructs an animatable full-head avatar from one or a few portraits while jointly estimating per-frame motion. The predicted asset is explicit: a neutral mesh, mesh-bound Gaussians, identity-adapted mesh blendshapes (MBA) and Gaussian appearance bases (GAB), so animation requires only basis mixing, skinning and rasterization. Two regularizers target what the authors call representation confounding: Neutral Identity Anchoring (NIA) applies an ArcFace identity loss to zero-expression renderings, and Adversarial Domain Alignment (ADA) discourages canonical vertex features from predicting dataset-level acquisition attributes. Experiments on VFHQ-Test, AVA256 and RenderMe-360 cover self/cross-reenactment, sparse-view reenactment and single-image full-head completion, with a public-data-only variant, a six-pair matched ablation, and desktop/mobile throughput measurements.

Critical strengths
  1. Extensive, protocol-matched and unusually transparent evaluation. Baselines are re-run under identical inputs, author-reported numbers are separated and excluded from ranking, raw and reference-relative AED are both reported, the registration step on RenderMe-360 is disclosed, and a public-data-only variant is trained with the same recipe. The public variant already improves VFHQ self-reenactment PSNR by 2.30 dB over FlexAvatar and raises cross-identity CSIM_Arc from 0.596 to 0.774 (Table 1), and it beats all baselines on every RenderMe-360 aggregate metric (Table 3). This substantially weakens the usual "gains come from private data" objection.

  2. The explicit asset is a practically important design and the efficiency claims are carefully scoped. Animation costs 0.04 GFLOPs versus 182.51 for FlexAvatar (Table 5a); four concurrent avatars reach 84 and 68 FPS at 1024×1024 on two phones. The paper scopes these numbers honestly as animation-and-rendering throughput with precomputed controls.

Critical weaknesses
  1. The qualitative evidence is thin relative to the extensive quantitative evaluation, and its baseline coverage does not match the tables. In the main paper, Fig. 3 shows two cross-reenactment examples per dataset, and the appendix adds two or three per figure (Figs. 7–9), so it is hard to judge at a glance how the methods differ. FastAvatar is reproduced in Tables 1, 2 and 5 and plotted in Fig. 1 but never appears in a qualitative figure; GAGAvatar appears in Fig. 7 but only with author-reported numbers in Table 1; LAM, the closest explicit-rig one-shot system (public code; reported AKD 2.06 and self CSIM 0.822), appears in neither the qualitative figures nor Table 5. On AVA256, FastAvatar's CSIM (0.170) and AKD (27.87 px), and UIKA's AKD (28.49) with visibly mis-posed, differently framed heads in Figs. 3(b) and 8, suggest that these pose-free methods may not be consuming the calibrated target cameras, in which case FlexAvatar is the main informative feed-forward competitor there and a footnote would help readers. More examples per comparison, FastAvatar outputs, GAGAvatar under the shared protocol, and, if feasible, LAM would bring the qualitative story in line with the quantitative one.

  2. End-to-end real-time, video-driven animation on device is not demonstrated. The mobile numbers (84 / 68 FPS for four avatars) are animation-and-rendering throughput with precomputed driving controls, measured with up to eight frames in flight, and the paper states that its own expression extraction is not yet real time on mobile (Sec. 4.7). The telepresence scenario that motivates the paper is therefore only partly shown. Per-frame latency (the component times in Table 5(b) exceed the totals), the cost of motion estimation and asset generation, asset size and peak memory would let readers judge how far the system is from an on-device pipeline; a demonstration with platform-native controls (e.g. ARKit) would also close much of the gap.

Questions for authors
  1. Attribution of the gains. In Table 4(a), the variant without NIA (A1) already reaches 24.79 dB / 0.095 LPIPS on VFHQ versus 24.98 / 0.093 for the full model, and removing ADA (A4) changes VFHQ by at most 0.06 dB, so most of the 2.5 dB gain over FlexAvatar seems to come from the base system (joint estimation, A0→A3: 20.54→24.72 dB; an 878M-parameter DINOv2-L backbone trained on sixteen 96 GB GPUs; the training corpus). NIA's measurable effect is cross-identity CSIM_Arc 0.680→0.777 with a small expression cost (cross-identity AED 0.230→0.240), and ADA's effect is confined to RenderMe-360. What are FlexAvatar's parameter count, training data and compute relative to yours, and what does the base architecture achieve before NIA, ADA, GAB and MBA are added? Do the NIA and ADA effects also hold in the Public setting, given that all ablations use Full? Will the public-data weights and code be released?

  2. RenderMe-360 protocol and the ADA mechanism. Full-head completion uses head masks from four additional calibrated views to fit a per-identity similarity transform, and the 80 registration views are part of the 1,180-view aggregate (Appendix A.6.2); the protocol is applied consistently to all methods, but it is no longer strictly single-image and the transform depends on the predicted geometry. Could you report Tables 3 and 4(b) with those views excluded, and A4 vs A5 under an identical registration transform? ADA improves frontal LPIPS by 13.5% (0.171→0.148, Table 4b), as much as rear-side (10.3%) and strict-rear (11.4%); how do you interpret this, and what is the accuracy of an independently trained probe for the three acquisition attributes on frozen canonical features with and without ADA?

  3. Expression fidelity. The full model has higher raw cross-identity AED (0.233 vs 0.221) and APD (0.029 vs 0.025) than FlexAvatar; Ref-AED removes per-trajectory constant offsets but would also hide a uniformly damped expression amplitude, which identity anchoring and identity-scaled blendshapes (MBA) could in principle induce. What is the ratio of predicted to driving expression magnitude with and without NIA and with and without MBA, and does the NIA identity gain persist under a non-face-recognition identity measure or a small user study, given that the training loss and the CSIM evaluators are all face-recognition embeddings? Per-identity variance for AVA256 (8 identities) and RenderMe-360 (20 identities) would also help.

3 comments
AuthorsOpenReview ↗

Response to Q1, Q2

Thank you for the detailed review and for recognizing the breadth of our evaluation and the practical value of the explicit avatar representation.

Q1

Model scale and training resources

We acknowledge that model scale and training resources may contribute to the overall performance gap. Our model has 878M trainable parameters, compared with FlexAvatar’s 269M total and 182M trainable parameters. Training takes approximately two weeks on 16 H20 GPUs, compared with three weeks on one A100 for FlexAvatar. Public shares several major training datasets with FlexAvatar, while Full adds synthetic and studio data. We have added the resource and dataset comparison in Appendix A.5 (Table 7).

The larger model generates an explicit avatar for efficient real-time animation and rendering on mobile devices. FlexAvatar’s animation requires a Transformer-based decoder, resulting in higher runtime costs. We also report inference costs in Appendix A.7 (Table 9): single-image avatar generation takes only 0.332 s on an RTX 4090.

Attribution of the gains and ablation design

Much of the reconstruction improvement comes from joint avatar–motion learning. In the matched A0–A3 comparison, PSNR increases from 20.54 to 24.72 dB and LPIPS decreases from 0.134 to 0.097. NIA addresses identity preservation during cross-identity animation, where small changes in facial features and motion can noticeably affect perceived identity. In the matched A1–A3 comparison, NIA raises cross-identity CSIM_Arc from 0.680 to 0.777. ADA improves full-head reconstruction, reducing rear-side and strict-rear LPIPS by 10.3% and 11.3%. Our ablations isolate these factors through matched comparisons; an all-components-removed baseline is not included.

NIA and ADA in the Public setting

Public achieves broadly comparable performance to Full on VFHQ and AVA256 and outperforms all baselines on the aggregate RenderMe-360 metrics. These results support the overall Public model, although matched Public ablations for NIA and ADA have not been completed.

Code and model release

The appendix provides details of our architecture, training, preprocessing, and evaluation protocols to support reproducibility. The release of the code and Public model weights remains under consideration.

Q2

Evaluation without registration views

We have removed the four registration cameras (24, 26, 28, and 55) from evaluation and updated Tables 3 and 4(b) using the remaining 1,100 views across 20 subjects. Full remains best across all five view groups and three metrics, and Public outperforms all baselines on the aggregate metrics. The ADA ablation conclusions also hold. The four head masks remain part of registration, as clarified in Appendix A.6.2.

Registration and frontal-view improvements

A4 and A5 use the same fitting procedure, with a transform estimated for each predicted geometry. Our comparison therefore evaluates reconstruction quality after registration.

The frontal set includes oblique views and cameras above and below the subject. In A4, we observe forward-leaning neck geometry and insufficient shoulder coverage, which cause silhouette and alignment mismatches in these views. Improvements in these regions can therefore benefit frontal LPIPS as well as rear-view reconstruction. We have added this analysis and corresponding qualitative comparisons in Appendix A.8.2 (Fig. 11).

ADA objective and acquisition-attribute predictability

We have not conducted an independent probe evaluation. ADA is designed to reduce reliance on acquisition-specific cues in mixed-source training. Our matched ablations show reconstruction improvements across viewing regions, supporting ADA’s benefit for full-head reconstruction.

AuthorsOpenReview ↗

Response to Q3

Q3

Identity preservation beyond face-recognition embeddings

We added an independent geometry-based evaluation of 50 VFHQ cross-identity sequences using RealDenseFace (Li et al., 2026), a state-of-the-art monocular 3D face reconstruction method. We track each generated sequence independently with a shared FLAME shape and compare the generated and source neutral meshes over 1,787 corresponding facial vertices in the FLAME model coordinate system, with pose and expression set to zero.

In the matched A1–A3 comparison, NIA reduces neutral-face RMSE by 12.00%, with improvements in 39 of 50 sequences. This supports better preservation of source facial geometry beyond CSIM. We also added eight comparisons of predicted neutral heads (Fig. 13). Without NIA, the predicted neutral heads exhibit noticeable changes in face, mouth, and eye shape, suggesting that identity-specific features leak into the expression representation.

Expression and pose fidelity

Full has higher raw cross-identity AED and APD than FlexAvatar under the evaluation used in Table 1. Ref-AED removes constant per-sequence offsets and measures relative expression changes, but it does not establish absolute expression accuracy or rule out reduced expression amplitude.

We additionally evaluate expression and head-pose accuracy using parameters tracked by RealDenseFace. Full achieves lower FLAME-AED than FlexAvatar (0.275656 vs. 0.292315), while its pose error remains higher (Table 10). The two expression evaluators give different rankings, so the results do not support a consistent expression-accuracy advantage over FlexAvatar. We also measure expression magnitude directly to examine the amplitude concern, as detailed below.

Expression magnitude with NIA and MBA

In the matched A1–A3 comparison, raw AED increases from 0.230 to 0.240 in Table 4(a), while FLAME-AED remains nearly unchanged under RealDenseFace (0.276953 vs. 0.277396; Table 10). The predicted-to-driving expression magnitude ratio decreases from 0.935 to 0.894, corresponding to a 4.36% reduction in mean expression magnitude. The expanded A1–A3 qualitative comparisons (Fig. 12) help interpret this reduction: A1 can produce exaggerated and unnatural mouth shapes, which NIA helps reduce in these examples.

Adding MBA in the matched A3–A4 comparison changes the magnitude ratio from 0.894 to 0.888, a further 0.69% reduction. Tables 4(a), 10, and 11 report these complementary measurements.

AuthorsOpenReview ↗

Response to Weakness

Weakness 1

Expanded qualitative evidence

We have expanded the qualitative comparisons to seven VFHQ examples and eight AVA256 examples (Figs. 7–8). The VFHQ comparison now includes GAGAvatar, FastAvatar, and LAM, and the AVA256 comparison includes FastAvatar. We also expanded the NIA ablation to four examples, added eight comparisons of predicted neutral heads, and included four full-head animation examples from photographic and stylized inputs, shown from front, side, and rear viewpoints.

The originally submitted supplementary video provides additional visual results, including comparisons with other methods and ablation results.

UIKA camera and alignment checks

We ran UIKA using its released weights, official preprocessing, and inference code with AVA256’s calibrated cameras and VHAP-tracked FLAME motion. On 336 valid target frames at 512 × 512 resolution, the tracked FLAME projections have a 41-point 2D landmark error of 4.98 pixels over the eyes, nose, and mouth, without additional 2D alignment. We also manually inspected a subset of frames and found the tracked FLAME projections reasonably aligned with the target images.

With the target camera and motion fixed, UIKA’s single-image prediction for the blue-haired subject in Fig. 3(b) has a 26.4% larger canonical head width and a 29.8% larger head height than its four-image prediction. This enlargement contributes to the misalignment in this example. In our qualitative inspection, similar enlargement occurred more often for subjects with long, loose hair, as illustrated in the additional examples.

FastAvatar camera setup

We used AVA256’s calibrated cameras and the FLAME parameters obtained through FastAvatar’s official code. Manual inspection showed reasonable alignment between the tracked FLAME projections and target images, with a 41-point 2D landmark error of 5.69 pixels, measured over the eyes, nose, and mouth across 393 valid frames at 512 × 512 resolution.

After carefully inspecting the released code, we found that it transforms Gaussian centers with head motion but leaves their orientations unchanged. We therefore switched to head-centric rendering, which slightly improved the visual results. Despite these efforts, we were unable to obtain satisfactory results on AVA256: self-reenactment AKD remains high at 21.06 and 21.93 pixels for single- and four-image inputs, respectively. We include these outputs in the qualitative comparisons (Fig. 8).

GAGAvatar and LAM comparisons

We reevaluated GAGAvatar under the shared VFHQ protocol using 2,500 frames per reenactment setting and updated Table 1. Full improves self-reenactment PSNR by 2.34 dB and cross-identity CSIM_Arc from 0.600 to 0.766 over GAGAvatar.

We also added LAM’s official implementation to the efficiency comparison in Table 5(a). On an RTX 4090 at 512 × 512 resolution, LAM takes 4.41 ms for animation and rendering and achieves 210.10 FPS with precomputed driving controls.

Weakness 2: End-to-end animation and system costs

Live on-device driving

We have added live camera-driven animation using MediaPipe expression and head-pose tracking on the same device, reported under “Live on-device driving” in the main text’s “Efficiency and Mobile Deployment” subsection. On iPhone 17, one avatar at 512×512 achieves 43.71 Hz tracking and a 59.99 Hz rendering loop over three 30-second runs, each following a 5-second warm-up. Latency from browser input sampling to the corresponding GPU completion notification averages 44.46 ms (p95: 61 ms), excluding sensor exposure and physical display latency. This demonstrates live on-device animation with MediaPipe controls; our own motion extractor remains non-real-time on mobile, and its acceleration remains future work.

Additional system costs

We added system-cost measurements on an RTX 4090 in Appendix A.7 (Table 9):

ComponentTimePeak GPU memory
Single-image avatar synthesis0.332 s4.65 GiB
Motion preprocessing16.18 ms/frame4.78 GiB (combined)
Motion network48.14 ms/frame
Animation and rendering—0.319 GiB

The runtime avatar asset occupies 167.71 MB before compression and 15.77 MB after compression. Table 9 also reports synthesis costs for 1–32 input images.

Mobile timing protocol

Table 5(b) measures animation and rendering throughput using precomputed driving controls. Component and total times are measured separately and are not additive; the reported frame times are not single-frame latency.

Reviewer_aJo7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes TaoHeadVerse, a feed-forward framework that reconstructs an animatable full-head avatar from one or a few portraits while jointly estimating motion. The predicted asset stores identity-adapted mesh blendshapes and Gaussian appearance bases. Its two key training constraints are Neutral Identity Anchoring (NIA), which supervises zero-expression renderings using identity features, and Adversarial Domain Alignment (ADA), which reduces acquisition-attribute predictability in canonical features for mixed-source full-head learning. Experiments on VFHQ, AVA256, and RenderMe-360 show improved reconstruction, identity preservation, and full-head completion.

Critical strengths
  1. This paper addresses an important practical problem: reconstructing explicit full-head avatars from one or a few portraits for expressive and identity-preserving animation and rendering on mobile devices.
  2. The experiments cover three datasets and include both Public and Full training variants. The strong Public results reduce concerns that the gains mainly come from private training data.
Critical weaknesses
  1. The designs of NIA and ADA are straightforward. NIA uses a frozen ArcFace encoder and an identity similarity loss, with the key design being supervision of zero-expression renderings. ADA follows a standard adversarial alignment approach, using an attribute classifier and a confusion loss to discourage canonical features from encoding dataset-level acquisition attributes. The novelty of this paper mainly in the problem formulation and system integration.
  2. The reported 60+ FPS on mobile devices requires precomputed driving controls. Since expression extraction is not yet real time on mobile, the full pipeline does not support end-to-end real-time animation, limiting its ability to meet the paper’s goal of live mobile avatar applications.
Questions for authors
  1. Could you report the RenderMe-360 results excluding the four registration views per subject? How much does the four-view registration affect the scores and method ranking? This would help clarify how much the evaluation depends on the additional multiview information.
  2. All component ablations use the Full training setting. Does ADA also improve full-head reconstruction in the Public setting?
2 comments

Thank you for the constructive feedback and for recognizing the practical value of explicit full-head avatars and the public-data results.

Q1: RenderMe-360 without registration views in scoring

Thank you for the suggestion. We have removed the four registration cameras (24, 26, 28, and 55) from evaluation and updated Tables 3 and 4(b) on the remaining 1,100 views across 20 subjects. For Full, aggregate LPIPS changes from 0.1746 to 0.1774, mask IoU from 0.9015 to 0.9002, and boundary F-score from 0.4210 to 0.4096. All method rankings remain unchanged across the five view groups and three metrics: Full ranks first in all 15 comparisons, and Public leads all baselines on the aggregate metrics. The ADA ablation conclusions also hold, with rear-side/strict-rear LPIPS reductions of 10.3%/11.3% relative to A4 (without ADA confusion). We have updated the protocol descriptions accordingly.

Four-view head-mask registration is still used by all methods; these results measure the effect of removing registration views from scoring.

Q2: ADA under Public training

We have not completed a matched ADA ablation under the Public setting within the rebuttal period. The Public model achieves broadly comparable reconstruction and reenactment performance to Full on VFHQ and AVA256 and outperforms all baselines on the aggregate RenderMe-360 metrics. The Public model retains strong performance across these datasets. The matched ADA ablations are reported in the Full setting.

Weakness 1: Contribution of NIA and ADA

Our contributions address identity leakage into the expression representation during joint avatar–motion learning and acquisition-related biases in mixed-source training. NIA applies identity supervision to zero-expression renderings, directly constraining the neutral asset. The DirectID control tests this choice against identity supervision of animated renderings: NIA improves cross-identity CSIM_Arc from 0.715 to 0.777, with comparable AED (0.242 versus 0.240). This supports the neutral-supervision design beyond adding a general identity loss. Figure 12 provides four qualitative comparisons of A1 (without NIA or DirectID), DirectID, and NIA.

Mixing monocular and multiview datasets introduces differences in acquisition conditions that can encourage source-specific predictions. ADA targets these biases through adversarial supervision of acquisition attributes in canonical features. The matched full-head ablation supports its reconstruction benefit across viewing regions, including rear-side and strict-rear LPIPS reductions of 10.3% and 11.3%. We combine these task-specific constraints with joint estimation and an explicit personalized avatar representation for efficient animation and rendering.

Weakness 2: Mobile throughput versus end-to-end animation

The original mobile throughput results use precomputed controls. We now additionally evaluate live driving with MediaPipe expression and head-pose tracking on the same device, reported under “Live on-device driving” in the main text’s “Efficiency and Mobile Deployment” subsection.

Using live camera input, one avatar at 512×512 on iPhone 17 achieves 43.71 Hz tracking and a 59.99 Hz rendering loop over three 30-second runs, each following a 5-second warm-up. Latency from browser input sampling to the corresponding GPU completion notification averages 44.46 ms (p95: 61 ms), excluding sensor exposure and physical display latency. This demonstrates live on-device animation using MediaPipe as the on-device tracker. Our own extractor remains non-real-time on mobile; we clarify this limitation and identify its acceleration as future work.

I thank the authors for their efforts to address my concerns. The rebuttal has clarified the issues I raised, and I remain positive about the paper.

Reviewer_dpTmOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proses a feed-forward method for 3DGS-based avatar creation from a single or few images. Fast inference time, even on mobile devices, is one of the main drivers for the methods design. Thus, instead of resorting to neural modifications needed during driving at inference, the authors design MAB and GAB mechanisms to adopt the blenshapes and Gaussians to become identity specific, s.t. inference can run with a minimal compute budget. As a result the paper achieves a inference speed up of 4.5x against their strongest baseline, FlexAvatar (Kirschstein et al.), and roughly 2x compared to UIKA. Another major focus area of the project is regularizing the training in such a way, that the loss is not simply reduced, but it is reduced by improving the desirable aspects of the pipeline, for example:

  • Since training is based on self-driving, preventing identity leaked through the expression branch is crucial. Their method seems to achieve this mainly via the blendshape design, but the authors introduce a variant of the common ArcFace loss, which they call NIA (Neutral Identity Anchoring). NIA is ablated to preserve identity better than the classical ArcFace loss, which they call DirectID, to the best of my understanding.
  • Another major challenge in feed-forward 3DGS avatar learning is the limited training data, since multi-view datasets are too small w.r.t. the identity and appearance/lighting axes. Thus mixing studio multi-view data with monocular in-the-wild video data is the most common remedy. Since those datasets follow extremely different distributions, the methods often start to cheat by adopting dataset specific behavior in the prediction of 3DGS. The paper proposes an adversarial stratgy, by training a discriminatior which needs to predict the dataset properties. The generator (i.e. the main 3dgs predictor model) needs to make the classification of the dataset properties as complicated as possible. An ablation shows that this especially helps with back-of-the head predictions.

Overall, the method is well-compared across several settings: monocular self-reenactment, 1-image and 4-image input novel-view self-reenactment, as well as cross reenactment for the same settings.

Critical strengths
  • Mainly I am impressed by the methods ability to learn from vastly different data sources. Maintaining complete predictions even in-the-wild-looking images at inference time is very tricky, as was shown in FlexAvatar (kirschstein et al.). Thus I bleieve that ADA is a superior approach to FlexAvatar's approach.
  • Furthermore, I appreciate the effort to make the avatars render efficiently, while I have to admit that MAB and GAB have a rather small effect, the base model without those additions still outperforms the baselines and is still fast. Due to the importance for many practical applications, this is an often overlooked component in research.
Critical weaknesses
  • To the community, it is not really clear what brings the bulk of the improvements. Especially, most of the ablation experiments only show minor differences (ignoring A0). Furthermore, the ablation is only conducted on monocular data, so the 3D aspect of the avatar only plays a minor role in the ablation. Ultimately, it alsmost seems like most of the improvements come from a combination of data preprocessing or dataset size, implementation details w.r.t. architecture, losses or training.
  • Overall, the novelty is slightly incremental, this the point above becomes even more crucial for the community. Neverthless, the results speak for the method and for its acceptance. I beleive that either additional ablations, or open sourcing training, preprocessing etc. would be important though, such that the research community can benefit from the main improvements of the paper.
Questions for authors
  • Why does the paper use ARKit? Why not FLAME, which is arguably more common? And Ablation A0 seems extremely in-transparent. What does fixed tracking mean? What tracking is used? is A0 traing from scratch with fixed tracking, or is it the same model, where the tracking is just replaced at inference time? In several places the paper highlights its design choice to learn the expression branch jointly, so a proper ablation would be important. Judging by the relatively bad results of A0, it seems that either the tracking was off, or there was another flaw in the experiment design. I would assume that, e.g. Pixel3dmm tracking, which e.g. FlexAvatar uses, would be a stronger baseline. Furthermore, such an ablation could really highlight the claims about the importance of joint training.
  • Are you sure that you have executed UIKA correctly? I ran it as a baseline for my own submission, and it seems to perform about as good as FlexAvatar for most tasks. Figure 3b) also shows severe misalginments. I am wondering why that is the case.
2 comments

Thank you for the thoughtful review and for recognizing the value of mixed-source learning and efficient avatar rendering.

Q1 The fixed-tracking ablation (A0) and the choice of ARKit

A0 is trained from scratch with offline motion estimates held fixed throughout training; it is not a jointly trained model with its tracking replaced at inference. We use Pixel3DMM for monocular videos and VHAP for multiview videos. For AVA256, FLAME is fitted to the official 3D landmarks. The fitted FLAME meshes are transferred to our ARKit representation through vertex fitting. Thus, A0 already uses Pixel3DMM, as suggested in your review.

A0 and A3 use matched initialization, training data, optimization protocols, evaluation targets, and architectures for the unaffected components. The controlled difference is fixed offline motion versus joint motion learning. We observe residual silhouette mismatch, particularly around hair and shoulders, which may contribute to A0's lower reconstruction scores. Joint learning allows motion estimates to adapt during avatar training. This comparison evaluates joint learning against the specified fixed-tracking pipeline, including the FLAME-to-ARKit transfer.

We chose ARKit blendshapes for their explicit and interpretable expression semantics, which provide named controls for animating the explicit avatar. The A0–A3 comparison uses the same representation in both variants. FLAME expression bases could also be used within this framework, although we have not evaluated this alternative.

Q2 UIKA alignment and evaluation

We ran UIKA using its released weights, official preprocessing, and inference code with AVA256’s calibrated cameras and VHAP-tracked FLAME motion. On 336 valid target frames at 512 × 512 resolution, the tracked FLAME projections have a 41-point 2D landmark error of 4.98 pixels over the eyes, nose, and mouth, without additional 2D alignment. We also manually inspected a subset of frames and found the tracked FLAME projections reasonably aligned with the target images.

With the target camera and motion fixed, UIKA’s single-image prediction for the blue-haired subject in Fig. 3(b) has a 26.4% larger canonical head width and a 29.8% larger head height than its four-image prediction. This enlargement contributes to the misalignment in this example. In our qualitative inspection, similar enlargement occurred more often for subjects with long, loose hair, as illustrated in the additional examples.

Weaknesses

Attribution of the gains

Joint avatar–motion learning yields the largest reconstruction improvement among our matched ablations. However, monocular reconstruction metrics alone do not fully characterize identity preservation during cross-identity animation or the quality of unseen head regions, which NIA and ADA specifically address. In the matched A0–A3 comparison, PSNR improves from 20.54 to 24.72 dB and LPIPS decreases from 0.134 to 0.097. NIA addresses source identity preservation during cross-identity animation. Small changes in facial features and expression can noticeably affect perceived identity, even when aggregate reconstruction metrics change little. In the matched A1–A3 comparison, NIA raises cross-identity CSIM_Arc from 0.680 to 0.777. ADA improves full-head reconstruction, reducing rear-side and strict-rear LPIPS by 10.3% and 11.3%, respectively.

Contribution and reproducibility

Our contributions address two specific problems in joint avatar–motion and mixed-source learning. NIA applies identity supervision to zero-expression renderings to limit identity leakage into the expression representation. Combining monocular and multiview datasets is common in avatar learning, but their acquisition differences can introduce source-specific biases. ADA discourages canonical features from relying on these cues. The DirectID control and matched ADA ablation support these design choices for identity preservation and full-head reconstruction, respectively. We combine these constraints with an explicit avatar representation that supports personalized basis mixing for efficient mobile animation and rendering.

GAB brings modest gains in aggregate metrics, but it captures expression-dependent facial appearance changes, such as the formation and relaxation of facial wrinkles. These effects are more apparent in motion, as shown in the GAB ablation videos in the supplementary material.

The appendix provides details of our architecture, training, preprocessing, and evaluation protocols to support reproducibility. The release of the code and Public model weights remains under consideration.

Three-dimensional evaluation of the ablations

Our multiview ablation focuses on reconstruction: Table 4(b) evaluates ADA on RenderMe-360 across frontal, rear-side, and strict-rear views, with qualitative comparisons in Fig. 14.

I thank the authors for the detailed rebuttal, which addresses most of my concerns. Thank you for pointing out subtleties of the contributions and metrics. Overall, I stand by my solid rating of 3. Unfortunately, I dont feel fully ready to raise my score to a 4 (the best score possible) due to my concerns about novelty.

Comments on the paper

1 comment

We thank all three reviewers for their careful reading and constructive feedback. We have revised the manuscript and added evaluations and clarifications addressing their main concerns:

  • RenderMe-360 evaluation: We updated Tables 3 and 4(b) using 1,100 evaluation views after removing the four registration cameras per subject.
  • Identity and expression evaluation: We added a geometry-based identity evaluation using RealDenseFace, a state-of-the-art monocular 3D face reconstruction method. This evaluates identity preservation through neutral-face geometry, independently of face-recognition embeddings. We also report tracked expression and pose errors on 50 VFHQ cross-identity sequences and expression-magnitude measurements for the NIA and MBA ablations.
  • Qualitative results: We expanded the VFHQ and AVA256 comparisons and the NIA ablation examples for cross-identity animation. We also added comparisons of predicted neutral heads with and without NIA, along with in-the-wild full-head animation examples.
  • Efficiency and deployment: We added live on-device driving with MediaPipe, LAM runtime measurements, and system costs covering avatar synthesis, motion estimation, asset size, and GPU memory.
  • Experimental details: We clarified the fixed-tracking ablation and added a comparison of model size, training datasets, and training resources with FlexAvatar.

We address the reviewers’ specific comments in separate replies.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.