acceptodds
Under review as a conference paper at ICLR 2027

One Basis to Animate Them All: Gaussian Blendshape Distillation for Real-Time Avatars

Abstract

3D Gaussian avatars support fast rendering, however, their real-time animation is often challenged by the costly neural inference. We address this bottleneck and show that the animation of pretrained avatar models can be closely approximated by a linear combination of identity-independent blendshapes. Building on this finding, we introduce GALA (Gaussian Animation via Linear Approximation), a distillation method that replaces per-frame heavy neural decoding with a shallow coefficient predictor and a linear blend. To improve fidelity and reduce memory requirements, we propose to construct the basis using block-local PCA under a rendering-aware metric and a memory budget. Our method learns a shallow MLP network to predict blendshape coefficients and applies to various animation architectures without retraining original models. We validate GALA by accelerating the inference of three distinct avatar models for 3D animation of facial expressions and full-bodies with clothing dynamics. Across these models, our distillation generalizes to held-out identities and reduces CPU animation cost by up to three orders of magnitude while preserving most of the rendering quality. Excellent results of our method confirm the shared linear structure of learned avatar representations and enable highly efficient and accurate animation at frame rates reaching up to 60fps on mobile devices. \em Our code and models will become publicly available.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3d gaussian splatting, linear structure, mobile, real-time, avatars, linear, models, blendshape, body, head

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_TocYOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses the problem of costly neural inference during real-time animation of 3D Gaussian avatars. The method aims to fill the gap between 3D morphable models, per-person blendshapes, fixed-rig avatars, and animatable avatar models by combining photorealism, identity-shared representations, and linear blendshape-based animation at runtime. The method extracts an identity-shared blendshape basis from pretrained avatar models using block-local, rendering-aware PCA with memory-budgeted component allocation. A lightweight coefficient predictor MLP replaces the costly neural decoder during animation. The method is evaluated on three different avatar architectures (AGORA, FlexAvatar, DynaAvatar) and achieves significant improvements in CPU animation speed while largely preserving rendering quality.

Critical strengths
  1. [Clear motivation] The paper addresses a valid and practically relevant problem. The idea of extracting an identity-shared linear blendshape representation from pretrained nonlinear avatar models is well motivated.
  2. Components are well thought shown their impact clearly in Table 3 and 4.
  3. [Practical Method] The method does not require retraining the original host models and enables real-time animation on mobile devices. The demo shows the approach is useful in practice.
Critical weaknesses
  1. [Limited novelty] Previous works such as GEM (Zielonka et al., 2025) and SqueezeMe (Iandola et al., 2025), and RGBAvatar have already explored distilling neural Gaussian avatars into efficient linear representations. Although the method introduces an identity-shared basis, the paper does not directly demonstrate its advantages over subject-specific distillation approaches. A controlled comparison in terms of reconstruction quality, memory usage and computational cost would help establish the contribution of the method.
Questions for authors

Could the authors provide a comparison between the identity-shared basis and a subject-specific PCA distillation baseline, similar to GEM, using the same host model? In particular, what is the trade-off between reconstruction quality, speed, and total memory requirements?

2 comments
AuthorsOpenReview ↗

Author Response to Reviewer TocY

Thank you for the positive review. The new results are included in the revised paper, where all changes are marked in blue.

W1. Novelty and comparison with subject-specific methods. We would like to respectfully clarify how GALA differs from GEM, SqueezeMe and RGBAvatar. These methods build a basis for one person from a multi-view or monocular capture of this person. In contrast, GALA takes a pretrained avatar network of many identities and distills the network itself into a single basis. This basis is shared by all identities of the network, including unseen people. To the best of our knowledge, such a distillation has not been attempted before.

Thank you for the suggestion to directly demonstrate the advantages over subject-specific methods. We agree that such a comparison is important. The original submission already compares GALA with GEM and RGBAvatar in App. H (Tab. 9 and 10). In cross-reenactment, FlexAvatar + GALA achieves lower expression and pose errors than all per-subject methods on INSTA, Ava256 and VFHQ, although FlexAvatar + GALA uses only a single photo of the subject. Since these methods use their own training pipelines, we have now added a controlled comparison in which both approaches use the same host model (Q1).

Q1. Shared vs. subject-specific basis with the same host model. Table R1 shows that, for unseen motion, the PSNR of our shared model is within 1 dB of the PSNR of the per-subject models, even though the test subjects are not used for the construction of the shared basis. For each test subject, we construct a per-subject basis and train a per-subject coefficient network using 512 outputs of the host model for this subject. We use the same host model, pipeline and memory budget as for the shared model, and we sample the animation parameters from the training data of the host model. For the head models, the per-subject models have a large advantage on their own training samples. Most of this advantage does not transfer to unseen motion, which indicates that the per-subject models overfit to their training samples. In cross-reenactment, which is the most realistic scenario, the shared and the per-subject models perform on par. The per-subject models are better for AGORA and the shared model is better for FlexAvatar, in both cases by about 0.6 dB. For DynaAvatar, we animate the test subjects with four motion sequences of other held-out subjects, and the difference between the shared and the per-subject models is only 0.01 dB. Given the same memory budget, the shared and the per-subject models also run at about the same speed. In contrast, the costs of the two approaches differ significantly. The storage of the shared basis is constant, whereas the storage of per-subject bases grows linearly with the number of identities and is 100 times larger for 100 identities (Table R1). Initializing a new identity takes seconds of GPU time with the shared model and minutes with a per-subject model. The reason is that the shared model requires a single inference of the host model, whereas per-subject models require many inferences and a PCA decomposition. The one-time construction of the shared model pays off after 30–50 identities. This comparison is now included in App. K of the revised paper.

Table R1. PSNR with respect to the host model, using predicted coefficients. In-train: training samples of each model, which belong to other identities for the shared model. Held-out: unseen motion of 10, 10 and 2 test subjects for AGORA, FlexAvatar and DynaAvatar. Cross-reenactment: motion of other people. The shared model is constructed once, which takes 80 min, 66 min and 16 h for the three host models. Bold: lower cost, and PSNR that is better by more than 0.1 dB.

basis storage, 100 identitiesGPU time, new identityin-trainheld-outcross-reenactment
AGORAshared0.05 GB<1 s30.6631.4131.40
per-subject4.8 GB1.9 min36.2532.3831.95
FlexAvatarshared0.05 GB2 s24.8326.8622.77
per-subject4.8 GB1.4 min33.6126.8222.14
DynaAvatarshared0.09 GB7 s27.4322.6523.13
per-subject8.8 GB29 min27.8922.9023.14
Reviewer_TocYOpenReview ↗

My points are addressed. I'll keep my positive rating.

Dear Authors, thank you for your detailed answers!

W1. Thanks for the clarification, and sorry that I missed the figure in Appendix H. Q1. Thanks for the detailed follow-up comparison.

I'll keep my positive rating

Reviewer_X9HqOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GALA, a method to distill pretrained Gaussian avatar models into a fixed blendshape basis and a small coefficient network. The basis is extracted from multiple identities and reused for new identities, while the coefficients depend on identity and animation inputs. The method combines per-attribute decomposition, local bases, rendering-aware PCA, and memory-based rank allocation. Experiments on different host models show reduced animation cost with some quality loss.

Critical strengths

S1. The shared basis across identities is a useful direction. Compared with subject-specific Gaussian blendshape methods, extracting one basis and reusing it for new identities could reduce repeated fitting and storage and improve computation efficiency.

S2. The evaluation covers three different host models. These include head avatars and full-body avatars with clothing dynamics. The fixed-memory ablations and quality-latency comparisons provide useful information about the representation and its computational benefit.

Critical weaknesses

W1. The technical novelty over GEM is limited. GEM [1] already distills Gaussian animation into per-attribute PCA bases and predicts coefficients with a regressor. GALA follows a similar framework, with the main difference being a basis shared across identities. Local bases also have precedents in WebAvatar [2]. The overall method appears to be an extension of existing Gaussian blendshape distillation rather than a substantially new approach.

[1] Zielonka et al. Gaussian Eigen Models for Human Heads. CVPR 2025.

[2] Zhan et al. High-Fidelity Mobile Avatars with Pruned Local Blendshapes. CVPR 2026.

W2. The experiments do not sufficiently validate the benefit of sharing the basis across identities. There is no controlled comparison with subject-specific PCA using the same teacher, data, and memory budget. The comparisons with existing per-subject methods use different training and reconstruction pipelines, so they cannot isolate this contribution.

W3. The FlexAvatar and DynaAvatar evaluations do not fully separate the method’s contribution from other factors.

FlexAvatar applies a neutral-avatar correction and 200 steps of coefficient-network fine-tuning to each test identity. Thus, it cannot valiadted the proposed framework role since the model has finetuned on each unseen identity. On Ava256, the host and GALA are evaluated on eight subjects, while competing results are quoted from a different test subset. This supports the paired host–GALA comparison, but limits the cross-method ranking.

For DynaAvatar, the method retains the host’s neutral representation and LBS, while evaluation mainly uses whole-image PSNR, SSIM, and LPIPS. The experiments do not isolate the contribution of the predicted dynamic residuals beyond the retained neutral avatar and LBS. A neutral-plus-LBS baseline can potentially achieve a similar result.

Therefore, the experiments and setting on FlexAvatar and DynaAvatar are not fair such that the proposed framework cannot be fully verified.

Questions for authors

Q1. Shared versus subject-specific bases: A comparison under the same teacher and memory budget would clarify the quality cost of sharing. Total storage and initialization cost for multiple identities would clarify its practical benefit.

Q2. FlexAvatar evaluation: Results before and after enrollment would separate direct generalization from per-subject adaptation. The Ava256 cross-method comparison also needs a common evaluation subset.

Q3. DynaAvatar dynamics: A quantitative neutral-plus-LBS baseline, together with evaluation on dynamic garment regions or temporal changes, would clarify how much of the host’s clothing dynamics GALA preserves.

Q4. Training data: Section 4.1 and Table 6 report different training identity counts for FlexAvatar. The exact samples used for basis extraction, coefficient training, and model selection need clarification. For DynaAvatar, distinct people and source-image identity instances should be counted separately.

Q5. Number of training identities: Results with different numbers of identities used for basis construction would clarify how much identity diversity is needed for the shared basis to generalize to unseen people.

3 comments
AuthorsOpenReview ↗

Author Response to Reviewer X9Hq (Part 1)

Thank you for the detailed review. We ran the requested control experiments and added them to the revised paper, where all changes are marked in blue.

W1. Novelty. We respectfully disagree that the novelty over GEM is limited, for two reasons.

The first reason is the problem that we address. GEM and WebAvatar build a basis for one person from a multi-view capture of this person, and GEM requires 27.8 h per subject (Tab. 9). In contrast, GALA takes a pretrained avatar network of many identities and distills the network itself into a single basis. The same basis is then reused for unseen people, without any capture and without a per-person basis. To the best of our knowledge, such a distillation has not been attempted before. It was also far from clear that such a shared basis exists, because nothing in the architecture or in the training of such avatar networks enforces a shared linear structure. One of the contributions of our work is that we show that such a basis exists for three very different host models, and we consider this finding the main result of the paper.

The second reason is the method. Simple PCA does not yield such a basis. With projected coefficients, a basis obtained with simple PCA is 10–13 dB less accurate than our basis for the same memory budget (Tab. 3a). The shared basis becomes accurate only with local per-attribute bases, which have precedents in GEM and WebAvatar, and with our PCA in a rendering-aware metric. This metric makes the components of all attributes comparable, which allows us to allocate the components according to their effect on the image. In contrast, allocating the components according to their variance, which is how WebAvatar prunes its local blendshapes, reduces the accuracy by up to 13 dB (Tab. 3a). The rendering-aware metric and the allocation are the core of our method, and they are neither standard nor obvious. Since GALA requires only the outputs of the host model, it can also be applied to other animatable Gaussian avatar models.

W2, Q1. Shared vs. per-subject basis. Table R1 shows that, for unseen motion, the PSNR of our shared model is within 1 dB of the PSNR of the per-subject models, even though the test subjects are not used for the construction of the shared basis. For each test subject, we construct a per-subject basis and train a per-subject coefficient network using 512 outputs of the host model for this subject. We use the same host model, pipeline and memory budget as for the shared model, and we sample the animation parameters from the training data of the host model. For the head models, the per-subject models have a large advantage on their own training samples. Most of this advantage does not transfer to unseen motion, which indicates that the per-subject models overfit to their training samples. In cross-reenactment, which is the most realistic scenario, the shared and the per-subject models perform on par. The per-subject models are better for AGORA and the shared model is better for FlexAvatar, in both cases by about 0.6 dB. For DynaAvatar, we animate the test subjects with four motion sequences of other held-out subjects, and the difference between the shared and the per-subject models is only 0.01 dB. In contrast, the costs of the two approaches differ significantly. The storage of the shared basis is constant, whereas the storage of per-subject bases grows linearly with the number of identities and is 100 times larger for 100 identities (Table R1). Initializing a new identity takes seconds of GPU time with the shared model and minutes with a per-subject model. The reason is that the shared model requires a single inference of the host model, whereas per-subject models require many inferences and a PCA decomposition. The one-time construction of the shared model pays off after 30–50 identities. This comparison is now included in App. K of the revised paper.

Table R1. PSNR with respect to the host model, using predicted coefficients. In-train: training samples of each model, which belong to other identities for the shared model. Held-out: unseen motion of 10, 10 and 2 test subjects for AGORA, FlexAvatar and DynaAvatar. Cross-reenactment: motion of other people. The shared model is constructed once, which takes 80 min, 66 min and 16 h for the three host models. Bold: lower cost, and PSNR that is better by more than 0.1 dB.

basis storage, 100 identitiesGPU time, new identityin-trainheld-outcross-reenactment
AGORAshared0.05 GB<1 s30.6631.4131.40
per-subject4.8 GB1.9 min36.2532.3831.95
FlexAvatarshared0.05 GB2 s24.8326.8622.77
per-subject4.8 GB1.4 min33.6126.8222.14
DynaAvatarshared0.09 GB7 s27.4322.6523.13
per-subject8.8 GB29 min27.8922.9023.14

W3, Q2–Q5. Please see our second comment.

AuthorsOpenReview ↗

Author Response to Reviewer X9Hq (Part 2)

W3, Q2. Enrollment. GALA also generalizes without enrollment (Table R2). Enrollment corrects the neutral avatar and fine-tunes the coefficient network. In PSNR, GALA is 0.5–1.7 dB below the host without enrollment and at most 1 dB below the host with enrollment. Visually, the difference from the host is mostly in lighting and color (Fig. 12 in App. L). Only FlexAvatar uses enrollment, because this host already fits an identity code to the input images of every test subject. Enrollment uses the same input images, never a target frame, and adds 4 s (about 7%) to the host's fit of about 1 min (Tab. 9). AGORA and DynaAvatar have no per-subject training (Tab. 6). Enrollment also never changes the shared basis and is not used in the ablations of Tab. 3 and 4. Table R2 and this discussion are now in App. L (referenced in Sec. 4.2).

Table R2. Tab. 2c with and without enrollment. All rows use the host's identity fit. Bold: the better GALA result.

VFHQ selfVFHQ crossAva256 single-shotAva256 few-shot
PSNR↑LPIPS↓AED↓CSIM↑AED↓APD↓PSNR↑AKD↓CSIM↑PSNR↑AKD↓CSIM↑
FlexAvatar23.290.0990.0750.6520.2260.02519.287.480.50821.506.170.755
+ GALA22.310.1090.0910.6800.2450.02919.257.520.49121.316.310.740
+ GALA w/o enrollment21.560.1320.1060.6340.2310.02618.757.990.43821.056.450.717

Ava256 subset. The test subsets differ only in the single-shot setting. The few-shot setting already uses a common subset. As in the FlexAvatar paper, this setting follows the Avat3r protocol, with the evaluation specification released for the Avat3r test identities (App. F). Our few-shot results are therefore directly comparable with the results of GPAvatar and Avat3r in Tab. 2c. In the single-shot setting, the FlexAvatar paper evaluates on 5 non-public subjects. We asked the authors to name these subjects before the submission but received no answer, so we use all 8 subjects of the released specification. The revised Tab. 2c marks the quoted single-shot results accordingly.

W3, Q3. Dynamics. This was a very useful suggestion. The Neutral+LBS baseline and the evaluation on dynamic regions both show that GALA preserves most of the host's clothing dynamics (Table R3). The baseline is GALA with all coefficients set to zero, which is the host's neutral avatar posed by skinning. Under the protocol of Tab. 2b, the predicted residuals close more than 60% of the PSNR gap between this baseline and the host, and the PSNR improves for all 83 sequences. In the dynamic regions, which are selected without using GALA, our model recovers 89% of the host's 3D motion beyond skinning and removes 70% of the image error of Neutral+LBS. Fig. 14 (App. M) shows where Neutral+LBS frequently fails: loose garments break into scattered Gaussians, whereas GALA follows the host. These results are now in App. M (referenced in Sec. 4.2).

Table R3. 83 test sequences. PSNR, SSIM and LPIPS against ground truth as in Tab. 2b. Recovery: the share of the squared error between Neutral+LBS and the host that a method removes. 3D: on the 10% most dynamic Gaussians. 2D: on the 10% of body pixels where Neutral+LBS and the host differ most.

PSNRSSIMLPIPS3D recovery2D recovery
DynaAvatar (host)23.860.9590.063100%100%
Neutral+LBS22.150.9540.071––
GALA23.220.9570.06688.9%70.2%

Q4. Training data. Thank you for catching this inconsistency. Sec. 4.1 still described an earlier data protocol. Tab. 6 is correct, so FlexAvatar uses 2,184 identities. For DynaAvatar, Table R4 counts the distinct people and the distinct outfits separately from the source images, as you asked. The revised paper corrects Sec. 4.1 and specifies the data for every stage in App. N (Table R4).

Table R4. Training data as identities × expressions per identity. For DynaAvatar, the first row gives source images × motion windows per image.

basis extractioncoefficient trainingmodel selection
AGORA10,000 × 49,500 × 6500 × 6
FlexAvatar2,184 × 4 to 292,184 × 1498 × 14
DynaAvatar662 × 201,035 × 2097 × 20
DynaAvatar, distinct people2614222
DynaAvatar, distinct outfits4215824

Q5. Number of training identities. For the head models, the accuracy of the basis saturates early (Fig. 15 in App. N). We built bases from subsets of the training identities and measured the PSNR to the host for held-out identities with projected coefficients. A basis from 256 identities is only about 0.7 dB below the full bases, which use 10,000 identities for AGORA and 2,184 identities for FlexAvatar. For DynaAvatar, the accuracy still improves by 0.5 dB from 16 to all 42 outfits. The limiting factor is the small amount of real data in 4D-Dress, not the method, and this shows the potential of the framework when more data become available.

AuthorsOpenReview ↗

Summary of Our Response to Reviewer X9Hq

Dear Reviewer X9Hq,

Thank you again for your review. As the discussion period is ending soon, we wanted to check whether our response and the revised paper address your concerns. For your convenience, here is a summary of our response to your comments W1–W3 and Q1–Q5:

W1. GEM and WebAvatar build a basis for one person from a capture of this person. In contrast, GALA distills a pretrained avatar network of many identities into a single basis that is reused for unseen people. Simple PCA does not yield such a shared basis, which becomes accurate only with our rendering-aware PCA and component allocation (see Tab. 3a).

W2, Q1. For unseen motion and in cross-reenactment, our shared basis is within 1 dB in PSNR of per-subject models, while significantly reducing storage and initialization costs (see Table R1, App. K).

W3, Q2. GALA generalizes without enrollment. Enrollment reduces the PSNR gap to the host and never changes the shared basis. Few-shot comparisons use a common subset, and the quoted single-shot results, which use a different test subset, are now marked (see Table R2, Fig. 12, App. L and the revised Tab. 2c).

W3, Q3. We added the Neutral+LBS baseline and the evaluation on dynamic regions that you suggested (Table R3 and Fig. 14, App. M). In the dynamic regions, GALA recovers about 90% of the host's 3D motion beyond skinning and removes about 70% of the image error of Neutral+LBS. Since our response, Table R3 has one more row with projected coefficients (see our reply to Reviewer Rjgh).

Q4. We corrected the FlexAvatar identity count and clarified training data for every stage, separating DynaAvatar people, outfits, and images (see Sec. 4.1, App. N and Table R4).

Q5. We added plots of the accuracy of the basis as a function of the number of training identities (Fig. 15, App. N).

If a concern remains open, please let us know which one, and we will respond promptly.

Reviewer_RjghOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes GALA, a method for accelerating pre-trained Gaussian avatar models by replacing the relatively expensive per-frame neural animation with a shared linear blend shape basis and a lightweight coefficient predictor. The basis represents each subject's animation residual relative to its own neutral pose and is shared across different identities within the same host model. The approach combines attribute-wise grouping with spatial local decomposition, a rendering-aware PCA, and memory-budget-constrained component allocation.

The authors evaluate the method on three host models, covering head generation, single-image head reconstruction, and full-body animation with dynamic clothing. The proposed method substantially accelerates animation computation on CPU while largely preserving the host models' rendering quality. They also demonstrate a mobile-browser demo.

Critical strengths
  1. The paper addresses an important distinction: efficient Gaussian rendering does not imply efficient neural animation. The reported speedups are impressive, and deploying neural avatars on resource-constrained devices sounds promising.

  2. Extracting a reusable animation basis from different identities, albeit a straightforward idea, is well motivated and demonstrated to be a challenging task; a set of novel techniques (attribute-wise grouping with spatial local decomposition, a rendering-aware PCA, and memory-budget-constrained component allocation.) need to be introduced to make the system work, which constitute and novelty of this paper.

  3. Experiments are done to three substantially different models, including a clothed human animation model conditioned on temporal information. The FPS boost is impressive. Ablation study is also solid.

Critical weaknesses
  1. The basis is shared across identities and evaluated on identities not involved in its construction, but FlexAvatar also uses a per-subject registration procedure to refine the neutral avatar and fine-tune the coefficient predictor. The paper does disclose this, but it should more clearly explain its implications in light of the generalization claims: which results demonstrate transfer with a non-adapted predictor, and which depend on the registration procedure?

  2. I find the qualitative results of cloth avatars is a bit underwhelming. This is expected as cloth deformation are non-rigid by nature so linear basis can never capture it faithfully. The host model (DynaAvatar) quality is also not very high; after all, it is a feedforward model while the non-rigidity of cloth deformation is much higher compared to that of head avatars.

Questions for authors
  1. What is the offline cost for each of metric estimation, basis construction, and coefficient-network training? Additionally, how long does FlexAvatar's per-subject registration procedure take?
4 comments
AuthorsOpenReview ↗

Author Response to Reviewer Rjgh

Thank you for the positive review and the helpful suggestions. The new results are included in the revised paper, where all changes are marked in blue. The tables are numbered consecutively across all our responses.

W1. Which results depend on the per-subject registration. Only FlexAvatar uses the per-subject registration, which is called enrollment in the paper. In the main paper, enrollment is used only for the FlexAvatar results in Tab. 2c and the FlexAvatar examples in Fig. 3. The ablations in Tab. 3 and 4 do not use enrollment. AGORA and DynaAvatar have no per-subject training (Tab. 6). All AGORA and DynaAvatar results therefore show transfer with a coefficient network that was never adapted to the test subject. FlexAvatar itself fits an identity code to the input images of every test subject, and enrollment uses only these images and never a target frame. Table R2 adds the FlexAvatar results without enrollment, i.e., without correcting the neutral avatar and without fine-tuning the coefficient network. GALA also generalizes in this setting. In PSNR, GALA is 0.5–1.7 dB below the host without enrollment and at most 1 dB below the host with enrollment. Fig. 12 in App. L compares the host, GALA and GALA without enrollment on an unseen VFHQ subject. Without enrollment, GALA still reproduces the expressions and the head pose of the host, and the remaining differences are mostly in lighting and color. Table R2 and this discussion are now in App. L (referenced in Sec. 4.2).

Table R2. Tab. 2c with and without enrollment. All rows use the host's identity fit. Bold: the better GALA result.

VFHQ selfVFHQ crossAva256 single-shotAva256 few-shot
PSNR↑LPIPS↓AED↓CSIM↑AED↓APD↓PSNR↑AKD↓CSIM↑PSNR↑AKD↓CSIM↑
FlexAvatar23.290.0990.0750.6520.2260.02519.287.480.50821.506.170.755
+ GALA22.310.1090.0910.6800.2450.02919.257.520.49121.316.310.740
+ GALA w/o enrollment21.560.1320.1060.6340.2310.02618.757.990.43821.056.450.717

W2. Cloth results. We agree that the quality of GALA is bounded by the quality of the host model, and that GALA inherits the limitations of the host model. Since GALA distills DynaAvatar, the clothing rendered by GALA cannot look better than the clothing rendered by DynaAvatar. However, GALA is not tied to this particular host model. When a better model for clothed avatars becomes available, our method can be applied to this model in the same way. Within the bound set by the host model, the linear basis is not the limiting factor. With projected coefficients, which show what the basis can express, our basis reproduces the renderings of DynaAvatar with a PSNR of 34.62 dB on held-out identities (Tab. 3a). With predicted coefficients, GALA also preserves most of the clothing dynamics of DynaAvatar. The new App. M (Table R3 in our second comment to Reviewer X9Hq) compares GALA with a baseline, the neutral avatar of the host model posed by skinning alone, and shows visual examples. In the dynamic regions, GALA recovers 89% of the 3D motion of the host model beyond skinning and removes 70% of the image error of this baseline.

Q1. Offline cost. Thank you for this question. Table R5 reports the GPU time of every stage and is now also included in App. O. In total, the three stages take slightly more than one hour for AGORA and for FlexAvatar, and about 16 h for DynaAvatar, where most of the time is spent on running the host model. These numbers show that the method is accessible. All stages run on a single GPU, and their cost is small compared with the cost of training the host models, which required 384 to 720 h of GPU time according to the original papers (first row of Table R5). For a new FlexAvatar subject, enrollment takes 4 s in addition to the identity fit of the host model, which takes about 1 min (Tab. 9).

Table R5. Offline cost of our method, measured as GPU time on one NVIDIA RTX 6000 Ada. Every stage is executed once per host model. The time for training the coefficient network includes the time for running the host model to generate the training targets, whereas Tab. 6 reports the time of the optimization alone. For comparison, the first row reports the training time of each host model as stated in the original paper, summed over all GPUs.

stageAGORAFlexAvatarDynaAvatar
host modeltraining384 h504 h720 h
GALA (ours)metric estimation20 min17 min1.5 h
basis construction37 min40 min8.5 h
coefficient-network training23 min9 min6.4 h
total1.3 h1.1 h16.3 h

Thanks for the detailed rebuttal. I am convinced about all issues except for the cloth avatar part. While the new table and qualitative results show improvement over a baseline, my original point was that "cloth deformation are non-rigid by nature so linear basis can never capture it faithfully". This was the central motivation why earlier works extended linear models (e.g., SMPL) for cloth modeling with non-linear neural nets [1,2,3 ]. The neutral+LBS baseline is rather weak as it is unclear how the LBS is constructed (kNN, or learned via approaches sush as [3]?), and even ignoring this baseline, the gap between GALA and the host model is real. With improved host model, GALA may have better overall results but the limitation of linear basis remains, and may become more obvious when the host model approximates real cloth dynamics better.

  1. Ma et. al. Learning to Dress 3D People in Generative Clothing, CVPR 2020
  2. Saito et. al. SCANimate: Weakly Supervised Learning of Skinned Clothed Avatar Networks, CVPR 2021
  3. Chen et. al. SNARF: Differentiable Forward Skinning for Animating Non-Rigid Neural Implicit Shapes, ICCV 2021
AuthorsOpenReview ↗

Author Response to Reviewer Rjgh (2)

Thank you for the quick reply. Your comment motivated us to investigate in more depth how much of the clothing dynamics our linear basis can capture.

We agree that cloth deformation is non-rigid and depends on the pose in a non-linear way. Our linear basis, however, does not make GALA linear in the pose. The basis is defined on top of a non-linear animation network: we construct the basis from the outputs of the host model, DynaAvatar, which maps the pose to the Gaussians with a neural network. We hypothesize that this network absorbs the non-linearity of the cloth deformation, and that the outputs of the network can be approximated well by a linear subspace. The basis spans this subspace. At run time, GALA replaces the host by a non-linear neural network that predicts the coefficients of the basis from the current pose and the 15 preceding poses, so that the dependence on the pose remains non-linear, as in [1–3].

We test this hypothesis without the coefficient network. On the 83 test sequences of held-out identities, we project the output of DynaAvatar onto the basis of 2,686 blendshapes (new last row of Table R3). In the dynamic regions, GALA with projected coefficients recovers 98.9% of the 3D motion of the host beyond skinning and removes 97.1% of the image error of Neutral+LBS. Against the ground truth, the PSNR of GALA with projected coefficients is 0.03 dB below the PSNR of the host. In the updated Fig. 14, GALA with projected coefficients is visually very close to the host on loose garments. The new row and the updated figure are in App. M of the revised paper. The linear subspace therefore loses very little of the clothing dynamics of DynaAvatar. Given this evidence, we believe that our method will also apply to future host models with more accurate clothing dynamics.

Table R3 with a new last row. 83 test sequences, with the metrics of App. M. Recovery is the share of the squared error between Neutral+LBS and the host that a method removes in the dynamic regions. The last row requires the output of the host at run time.

PSNRSSIMLPIPS3D recovery2D recovery
DynaAvatar (host)23.860.9590.063100%100%
Neutral+LBS22.150.9540.071––
GALA23.220.9570.06688.9%70.2%
GALA (projected)23.830.9590.06498.9%97.1%

The gap between GALA and the host. We agree that this gap is real. The rows GALA and GALA (projected) share the basis and differ only in the coefficients. Most of the gap therefore originates from the prediction of the coefficients, which we name as our first limitation (Sec. 5), and only a small part originates from the linear basis.

The Neutral+LBS baseline. Similar to [3], DynaAvatar uses a forward skinning field in canonical space. In DynaAvatar, the field is obtained by diffusing the SMPL-X skinning weights into a voxel grid, whereas [3] learns the field. In addition, the decoder of DynaAvatar adds a predicted offset to the skinning weights of the Gaussians at every frame. Neutral+LBS is GALA with all coefficients set to zero, i.e., the neutral avatar of the host, posed with the skinning weights of the neutral avatar at every frame. Neutral+LBS is thus the reference without time-varying attributes, and the recovery values are defined relative to this reference.

Better host models. For a host with more detailed garments, the basis would be constructed from the outputs of this host. Whether our hypothesis also holds for such a host cannot be tested with the hosts that exist today, and we state this limitation in App. M. The capacity of the basis is set by the memory budget, and for DynaAvatar the results with projected coefficients approach the host as the budget grows (Fig. 4).

[1] Ma et al. Learning to Dress 3D People in Generative Clothing, CVPR 2020

[2] Saito et al. SCANimate: Weakly Supervised Learning of Skinned Clothed Avatar Networks, CVPR 2021

[3] Chen et al. SNARF: Differentiable Forward Skinning for Animating Non-Rigid Neural Implicit Shapes, ICCV 2021

Thank you for your updated experiments. Now I am convinced that the linear basis is good enough to capture the cloth deformation given accurate enough per-frame coefficients; and since the coefficients are non-linearly predicted by a MLP, so cloth deformation is not, at least in theory, a bottleneck for GALA. I will remain positive on this paper.

Comments on the paper

1 comment

Dear Area Chair and Reviewers,

Thank you for the reviews and for the discussion. Reviewers TocY and Rjgh engaged in the discussion, and both wrote that our responses addressed their concerns and that they remain positive.

Reviewer X9Hq has not replied yet. We would like to note that most concerns of Reviewer X9Hq were also raised by the other two reviewers, and that we answered them with the same clarification and the same experiments. The novelty over GEM (W1) was also questioned by Reviewer TocY. The controlled comparison of our shared basis with per-subject bases for the same host model (W2, Q1) was also requested by Reviewer TocY (App. K). The dependence of the FlexAvatar results on enrollment (W3, Q2) was also raised by Reviewer Rjgh (App. L). The clothing dynamics of DynaAvatar (W3, Q3) were discussed with Reviewer Rjgh on the basis of the Neutral+LBS baseline that Reviewer X9Hq suggested (App. M). Only the Ava256 test subset (Q2) and the questions on the training data (Q4, Q5) are specific to Reviewer X9Hq. Our comment "Summary of Our Response to Reviewer X9Hq" covers every point of the review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.