acceptodds
Under review as a conference paper at ICLR 2027

Learning Animation-Aware 3D Gaussians with Native Rigging

Abstract

3D Gaussian Splatting (3DGS) has achieved remarkable performance in novel-view synthesis, yet directly converting static 3DGS into animatable assets remains challenging. Existing automatic rigging methods primarily target mesh-based representations, while large-scale rigged 3D Gaussian datasets are lacking. Moreover, even after rigging, Gaussians optimized for static rendering can produce artifacts during animation, particularly around joints and in poorly observed regions. To address these challenges, we present a framework that integrates automatic rigging with animation-aware Gaussian generation. Our method predicts skeletal structure and skinning weights directly from static 3DGS, then uses the predicted skeleton to guide Gaussian adaptation. This process refines Gaussian distributions and shapes to improve deformation robustness while preserving the original appearance. To support learning, we introduce a data construction pipeline that produces a large-scale paired dataset of vanilla and animation-aware Gaussian representations, together with skeletons and skinning weights. Extensive experiments demonstrate the effectiveness of our framework in automatic rigging and reducing deformation-induced rendering artifacts.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
3D Gaussian Splatting, Rigging, Animation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_QEphOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents a framework for converting static 3D Gaussian Splatting assets into rigged, animation-aware representations. It adapts a pretrained rigging model to predict skeletons and skinning weights directly from Gaussian inputs, and introduces a rig-guided adaptation module that refines Gaussians in approximately rigid regions while regenerating those in non-rigid regions. The authors also construct a large-scale paired dataset of vanilla and animation-aware Gaussians to supervise the framework.

Critical strengths

The paper identifies a meaningful limitation of animating static 3DGS: accurate rigging alone does not ensure artifact-free deformation, because the spatial support and distribution of the original Gaussians may be unsuitable for articulation. Jointly addressing rigging and representation adaptation is well motivated.

The two-branch adaptation strategy provides a sensible balance between appearance preservation and geometric flexibility. Restricting updates in approximately rigid regions while allowing regeneration near articulated regions is an intuitive design.

The paired dataset and its construction pipeline constitute a useful contribution. The reported quantitative results and qualitative comparisons provide encouraging evidence that animation-aware Gaussian adaptation can reduce deformation artifacts.

Critical weaknesses

The practical applicability and visual evidence need further development. The paper motivates converting existing static Gaussian assets into animatable content, but the demonstrated applications remain relatively narrow. The authors should clarify where this representation offers a practical advantage over established mesh-based animation pipelines, including those used in embodied simulation, and how it could complement video-based world-model workflows. This does not require demonstrating superiority across all these settings, but it does require a clearer account of the intended use cases and their specific benefits. More diverse application demonstrations would strengthen the contribution. In particular, static figures are insufficient to establish animation quality: demo videos showing continuous motion, multiple viewpoints, large articulations, and failure cases are essential for assessing temporal consistency and deformation artifacts.

The robustness of Gaussian adaptation on real captured assets remains insufficiently established. The principal training and evaluation assets are constructed from rigged meshes, while the generated-asset examples provide limited evidence of robustness to reconstruction noise, incomplete coverage, and uncertain rigging in practical inputs. I would encourage an evaluation on a real multiview human dataset, such as ZJU-MoCap, or a comparable setting. One informative protocol would reconstruct a canonical Gaussian asset using the multiview observations at a single timestamp, apply rigging and Gaussian adaptation, and evaluate its animation against held-out motion frames using PSNR, SSIM, and LPIPS. The comparison should include the unadapted reconstruction under the same rig and appropriate reconstruction or animation baselines, with supervision differences clearly specified. Reporting results with both estimated and reference rigs would additionally distinguish adaptation robustness from upstream rigging errors. Such an experiment would directly test whether the proposed refinement and regeneration remain effective beyond the mesh-derived training distribution.

Questions for authors

please refer to weakness.

4 comments
AuthorsOpenReview ↗

Response to Reviewer QEph

1. Practical advantages of native Gaussian rigging.

Our method is intended for static assets that are already represented as 3D Gaussians. This representation supports fast, photorealistic rendering and can be particularly useful for assets that may require additional material or hair modeling to render faithfully in a mesh-based pipeline, such as objects with glossy or furry surfaces. Recent methods, including DRiVE, G-Skin, RigGS, and GaussiAnimate, have explored rigging and animation with 3D Gaussians in different settings.

A mesh-based workflow for animating an existing Gaussian asset typically involves extracting a mesh, rigging it, and transferring the rig back to the Gaussians. Mesh extraction and rig transfer can both introduce errors. Our method predicts the skeleton and skinning weights directly from the input Gaussians, avoiding the need for either step. Gaussian adaptation further reduces artifacts that can arise when the Gaussian primitives deform under skeletal motion.

2. Video demo.

We have added a project page and included its link in both the abstract and original files in the supplementary material. The page features numerous video demonstrations and comparisons with existing methods, highlighting the advantages of our approach. Higher-resolution versions of the videos are available in the supplementary material and can be viewed locally without streaming delays.

3. Real captured human dataset.

Following the suggested experimental setup, we have added experiments on real-world human data (ZJU-Mocap and DDC) in Sec. 5 (lines 414-431). Starting from statically reconstructed 3D Gaussians, we predict a skeleton and estimate skinning weights for that skeleton. We then fit the rigged Gaussians to the dynamic sequences by optimizing the skeletal poses and evaluate our method by comparing the resulting renderings with the corresponding ground-truth video frames. Table 4 reports the quantitative results, and Fig. 6 shows the qualitative comparisons. The relatively small number of input views available for static Gaussian reconstruction (e.g., 23 camera views for ZJU-MoCap) can limit the quality of the initial reconstructions and, in turn, the quality of the rendered animations.

Table 4. Comparison of rendering quality on real-world datasets.

MethodZJU-MoCap PSNR ↑ZJU-MoCap SSIM ↑ZJU-MoCap LPIPS ↓DDC PSNR ↑DDC SSIM ↑DDC LPIPS ↓
RigAnything18.260.93710.082525.690.97820.0248
UniRig21.800.95140.051827.860.98200.0161
Puppeteer18.230.93820.080325.590.97870.0252
SkinTokens22.550.95380.045928.030.98240.0156
Ours22.880.95470.044328.090.98240.0154
Ours*22.950.95630.044928.140.98250.0160
AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer QEph,

Thank you again for your thoughtful review. As the discussion deadline approaches, we would appreciate knowing whether our response addressed your concerns or if any points still need clarification.

Best regards,

The Authors

Thanks to the authors for their comprehensive comments.

Most of my problems have been solved. Animation in Gaussian would be an important problem in VR/XR.

I would like to keep my initial score.

AuthorsOpenReview ↗

Official Comment by Authors

Thank you for taking the time to carefully review our rebuttal and provide your feedback. We are glad to hear that our responses have addressed your concerns. We appreciate your constructive comments and suggestions, which have helped us further improve the clarity and quality of our manuscript.

Reviewer_v7V1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper converts static 3D Gaussian assets into rigged, animation-aware representations. It adapts SkinTokens to Gaussian inputs and introduces a rig-guided, two-branch adaptation module: approximately rigid regions undergo restricted attribute refinement, while non-rigid regions are regenerated with greater freedom to change their spatial distribution and attributes.

Critical strengths
  • The distinction between rigging quality and deformation quality useful. Refining rigid regions while regenerating non-rigid ones is a reasonable way to address this problem, rather than only improving skeleton and skinning prediction.
  • The paired dataset supports further research on Gaussian rigging and adaptation.
  • The offline construction results in Table 3 are also encouraging, showing better animation quality without a large increase in Gaussian count.
Critical weaknesses
  • It is not clear how much of the reported improvement comes from the learned Gaussian adaptation. Table 3 evaluates offline construction, while Table 6 compares decoder variants. Neither provides a before/after adaptation comparison under the same skeleton and skinning function. The animation evaluation also uses GT skeletons, so I am unsure how well the full pipeline works with its own predicted rig.
  • The method aims to preserve the original appearance(second contribution bullet, L76), but it lacks a comparison between the input and adapted assets. The rigid branch adjusts scale, rotation, and opacity. The generation branch also discards higher-order SH. These changes may affect appearance, especially in regenerated regions, but their impact is not separately evaluated.
Questions for authors

Which rows in Table 2 include Gaussian adaptation? Could you clarify the Gaussian representations and the sources of skeletons and skinning weights in Tables 3, 5, and 6? A before/after comparison under the same skeleton, skinning function, and evaluation conditions would help show the improvement from learned adaptation.

When GT skeletons are used, are the skinning tokens predicted conditioned on them? How are weights assigned to newly generated Gaussian centers? How does animation quality change when both the skeleton and skinning are predicted by your method?

How much does the adapted asset differ from the input when rendered in the rest pose from held-out views? Could you report this for the whole object and regenerated regions, together with the fraction of input Gaussians replaced? Does discarding higher-order SH noticeably affect appearance in those regions?

4 comments
AuthorsOpenReview ↗

Response to Reviewer v7V1

1. Effectiveness of Gaussian adaptation

We have revised Table 2 to clearly distinguish the results obtained using the input Gaussians from those obtained after Gaussian adaptation. Both the baselines and Ours predict skinning weights directly from the input 3D Gaussians and the GT skeleton. Ours* represents the results of our full pipeline, which replaces the input Gaussians with the adapted 3D Gaussians while retaining the same skinning weights. Ours and Ours* clearly show the difference in rendering performance before and after 3D Gaussian adaptation.

Table 2. Comparison of rendering quality.

MethodPSNR ↑SSIM ↑LPIPS ↓
RigAnything24.160.95610.0599
UniRig27.070.97260.0334
Puppeteer29.220.97980.0230
SkinTokens29.690.98030.0238
Ours32.530.98680.0146
Ours*33.350.98760.0129

2. Appearance comparison between input and adapted 3D Gaussians on static pose.

We revised the Gaussian adaptation network so that it first predicts an offset for each input Gaussian and then generates additional Gaussians to supplement them, replacing the previous approach of fully regenerating those non-rigid regions. In the revised formulation, higher-order SH coefficients are retained. We also added a direct comparison between the input and adapted 3D Gaussians in the rest pose, which is the input pose. We render both representations from the same viewpoints and compare the resulting images with the corresponding ground-truth images. As shown in Table 3, compared with the input Gaussians, the adapted Gaussians achieve higher PSNR, with only a slight decrease in SSIM and a slight increase in LPIPS. Table 2 also shows that the adapted Gaussians yield higher rendering quality during animation than the input Gaussians.

Table 3. Comparison of rest-pose rendering quality between the input and adapted 3D Gaussians.

MethodPSNR ↑SSIM ↑LPIPS ↓
Input-GS43.890.99860.0019
Adapted-GS44.240.99830.0023

3. Performance of full animation pipeline.

We have added experiments of full animation pipeline on real-world human data (ZJU-Mocap and DDC) in Sec. 5 (lines 414-431). Starting from statically reconstructed 3D Gaussians, we predict a skeleton and estimate skinning weights for that skeleton. We then fit the rigged Gaussians to the dynamic sequences by optimizing the skeletal poses and evaluate our method by comparing the resulting renderings with the corresponding ground-truth video frames.

We further isolate the effect of Gaussian adaptation by comparing Ours, which uses the input Gaussians, with Ours*, which uses the adapted animation-aware Gaussians. Both configurations use the same predicted skeleton and skinning weights. Table 4 reports the quantitative results, and Fig. 6 shows the qualitative comparisons. The relatively small number of input views available for static Gaussian reconstruction (e.g., 23 camera views for ZJU-MoCap) can limit the quality of the initial reconstructions and, in turn, the quality of the rendered animations. Because Gaussian adaptation primarily refines local details, its benefits may be less apparent when the overall rendering quality is limited by artifacts already present in the input reconstruction.

Table 4. Comparison of rendering quality on real-world datasets.

MethodZJU-MoCap PSNR ↑ZJU-MoCap SSIM ↑ZJU-MoCap LPIPS ↓DDC PSNR ↑DDC SSIM ↑DDC LPIPS ↓
RigAnything18.260.93710.082525.690.97820.0248
UniRig21.800.95140.051827.860.98200.0161
Puppeteer18.230.93820.080325.590.97870.0252
SkinTokens22.550.95380.045928.030.98240.0156
Ours22.880.95470.044328.090.98240.0154
Ours*22.950.95630.044928.140.98250.0160

4. Rigging for adapted 3D Gaussians.

In Sec. 3.2 (lines 288–298), we have added a description of how the predicted skeleton and skinning weights are applied to the adapted 3D Gaussians. The skeleton is reused directly, while the skinning weights are transferred from the input Gaussians to the adapted Gaussians via k-nearest-neighbor interpolation in 3D space. The k-nearest-neighbor interpolation scheme in Eq. (3) can be used to transfer skinning weights between Gaussian representations of the same shape, even when they differ in the number or ordering of Gaussians.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer v7V1,

Thank you again for your thoughtful review. As the discussion deadline approaches, we would appreciate knowing whether our response addressed your concerns or if any points still need clarification.

Best regards, The Authors

Thank you for the clarification. I will take your response, along with the comments from other reviewers, into account for my final assessment.

AuthorsOpenReview ↗

Official Comment by Authors

Thank you for your response. To save you time, we briefly summarize below the revisions and additional experiments we have made to address your concerns:

  • Effectiveness of Gaussian adaptation: Direct comparisons of rendering quality before and after adaptation under the same skeleton and skinning field (Table 2).
  • Appearance preservation: Revised network architecture, retention of higher-order SH coefficients, and additional rest-pose rendering comparisons (Table 3).
  • Full animation pipeline: Additional experiments on ZJU-MoCap and DDC using predicted skeletons and skinning weights (Table 4 and Fig. 6).
  • Rig transfer from input to adapted Gaussians: Clarification of how the predicted skeleton and skinning weights are transferred to the adapted Gaussians (Sec. 3.2).

We would appreciate your feedback on whether these revisions and additional results have addressed your concerns. Please let us know if any specific issues remain unresolved or require further clarification.

Reviewer_a3zuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies auto-rigging problem for Gaussian Splatting based representations. It bases on an off-the-shelf rigging prediction model for meshes, and adapt it for predicting rigging parameters for Gaussian Splatting. It then performs an adaptation process to refine the Gaussians for better animation. It then constructs a dataset of animation-aware Gaussians for the network training. The paper evaluates skeleton and skinning prediction performance, the rendering quality, and ablation studies show the effectiveness of each module.

Critical strengths

The task studied in this paper is important and relevant, with a lot of applications in the animation industry. Although existing works have focuses on creating rigs for meshes, GS definitely has unique challenges when creating rigs for them. Therefore this paper is tackling an important yet challenging problem.

Critical weaknesses
  • The biggest concern for this paper is a lack of discussion with closely relevant baselines and related works. For example, DRiVE[1] also predicts rigging for Gaussian splatting, yet it is not cited or discussed. Similarly, G-Skin[2] is cited here but not being compared. Even if it's a per-instance optimization pipeline, it should still be compared. Dr. Robot [3] focuses on a different domain, but the core problem of binding rigging to a gaussian splatting is the same.
  • It remains unclear on whether the full animation pipeline produces the desired results. In Table 2, the skeleton used is GT skeleton, instead of predicted ones. Table 1 only evaluates the correctness on the rigging, but not on the full rendering and animation. Therefore, it is hard to infer whether the entire pipline (from GS to the final animation and rendering) is really working and is better than the baselines.

References.

[1] Sun et al. DRiVE: Diffusion-based Rigging Empowers Generation of Versatile and Expressive Characters. CVPR 2025.

[2] Yao et al. G-Skin: Learning to Bind 3D Gaussians with Generative Visual Priors. arXiv, August 2026.

[3] Liu et al. Differentiable Robot Rendering. arXiv, October 2024.

Questions for authors
  • How is the method compared to other relevant baselines?
  • What's the performance of the method for the entire pipeline evaluation?
5 comments
AuthorsOpenReview ↗

Response to Reviewer a3zu

1. Discussion and comparisons with closely relevant baselines.

Dr. Robot addresses a different task. Our method takes a single static 3D Gaussian representation as input, whereas Dr. Robot requires a known robot skeleton, multiple pose configurations, and corresponding multi-view images to perform 3D reconstruction and skinning optimization. Its skinning optimization relies on motion priors and pose supervision that our method does not require, making a direct comparison inappropriate. G-Skin's code is not publicly available.

We have added a discussion of and comparison with DRiVE. Note that DRiVE primarily focuses on humanoid characters, with its body-joint prediction model relying on a predefined set of 25 human-body joints. In contrast, our method supports arbitrary character categories and skeletal topologies, including both humans and animals.

Regarding the implementation, the officially released DRiVE code currently provides only the joint position prediction module, without the skeleton connectivity or skinning prediction modules. We therefore restrict our comparison to the J2J metric. Given DRiVE’s focus on humanoid characters, we evaluate both methods on the same subset of 451 humanoid characters from our test set. The mean J2J error is 7.09 for DRiVE and 1.88 for our method. We have added this comparison and the corresponding qualitative results to the revised manuscript (lines 384–390 and Fig. 9).

We also discuss Dr. Robot, and DRiVE in the Related Work section (lines 125–130).

2. Performance of full animation pipeline.

We have added experiments of full animation pipeline on real-world human data in Sec. 5 (lines 414-431). Starting from statically reconstructed 3D Gaussians, we predict a skeleton and estimate skinning weights for that skeleton. We then fit the rigged Gaussians to the dynamic sequences by optimizing the skeletal poses and evaluate our method by comparing the resulting renderings with the corresponding ground-truth video frames. Ours* denotes the results of our full pipeline. Table 4 reports the quantitative results, and Fig. 6 presents the qualitative comparisons. The relatively small number of input views available for static Gaussian reconstruction (e.g., 23 camera views for ZJU-MoCap) can limit the quality of the initial reconstructions and, in turn, the quality of the rendered animations.

Table 4. Comparison of rendering quality on real-world datasets.

MethodZJU-MoCap PSNR ↑ZJU-MoCap SSIM ↑ZJU-MoCap LPIPS ↓DDC PSNR ↑DDC SSIM ↑DDC LPIPS ↓
RigAnything18.260.93710.082525.690.97820.0248
UniRig21.800.95140.051827.860.98200.0161
Puppeteer18.230.93820.080325.590.97870.0252
SkinTokens22.550.95380.045928.030.98240.0156
Ours22.880.95470.044328.090.98240.0154
Ours*22.950.95630.044928.140.98250.0160
AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer a3zu,

Thank you again for your thoughtful review. As the discussion deadline approaches, we would appreciate knowing whether our response addressed your concerns or if any points still need clarification.

Best regards, The Authors

Thanks for your rebuttal! One question: why is the new full-animation experiment conducted on ZJUMoCap, a human capture dataset? It seems that this is a new dataset that has never appeared in the original text. If evaluation on ZJUMoCap, should it also compare against SotA neural avatar methods?

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response

We thank the reviewers for their constructive feedback. In the revised manuscript, we have expanded the experimental evaluation and clarified the contributions and practical scope of our method. Changes to the manuscript are highlighted in purple in the revised PDF. Below, we summarize the main revisions and our responses to the reviewers’ comments.

  • Discussion and comparisons with closely relevant baselines (a3zu). We have added a discussion of Dr. Robot and G-Skin, explaining why direct comparisons are not feasible. We have also added a comparison with DRiVE on the humanoid subset of our test set, as DRiVE was designed specifically for humanoid characters.
  • Performance of full animation pipeline and comparisons on real-captured human data (a3zu & v7V1 & QEph). We have added experiments to evaluate our full animation pipeline against the baselines on two real-world human motion capture datasets, ZJU-MoCap and DDC. Table 4 and Fig. 6 report the quantitative and qualitative comparisons.
  • Effectiveness of Gaussian adaptation (v7V1). We have revised parts of the Gaussian adaptation network architecture and added direct comparisons of rendering quality before and after Gaussian adaptation, both in the rest pose and in deformed poses. Tables 2,3,4 and Figs. 5,6,10 report the corresponding additional results.
  • Rigging for adapted 3D Gaussians (v7V1). We have added technical details on how to transfer predicted rigging, specifically skinning weights, from the input Gaussians to the adapted 3D Gaussians.
  • Practical advantages of native Gaussian rigging (QEph). We further elaborate on the practical utility and core objectives of our work.
  • Video demo (QEph). We have added a project page with video comparisons of the animation results. Please see the anonymous link in the abstract or the files in the supplementary material.

Dear Reviewers,

Thank you for your time and effort in reviewing our submission. We have provided detailed responses to each concern raised. As the Reviewer-Author discussion period is drawing to a close (less than 12 hours), could you please let us know whether your concerns have been clearly addressed? We look forward to your further feedback.

Best regards,

The authors

AuthorsOpenReview ↗

Rebuttal Summary for the Area Chair

Dear AC,

To facilitate your assessment, we provide a brief summary of the reviewers’ concerns, our revisions, and the outcomes of the rebuttal discussion. We sincerely thank all reviewers for their time and constructive feedback.

Summary of Concerns and Corresponding Revisions

  • Comparisons with closely related methods (a3zu): We discussed Dr. Robot and G-Skin, clarified why direct comparisons are not feasible, and added a comparison with DRiVE on the humanoid subset.
  • Full animation pipeline and real-world evaluation (a3zu, v7V1, QEph): We added experiments on ZJU-MoCap and DDC, evaluating our full pipeline against existing baselines (Table 4 and Fig. 6).
  • Effectiveness of Gaussian adaptation and appearance preservation (v7V1): We revised the network architecture and added direct comparisons before and after adaptation in both rest and deformed poses (Tables 2–4 and Figs. 5, 6, and 10).
  • Rigging transfer to adapted Gaussians (v7V1): We provided additional technical details on transferring the predicted rig, particularly skinning weights, to the adapted Gaussians.
  • Practical applicability and visual evidence (QEph): We clarified the practical advantages of native Gaussian rigging and added a project page with extensive video demonstrations and comparisons.

Outcomes of the Reviewer Discussion

  • Reviewer QEph confirmed that our responses addressed their concerns and maintained their positive rating.
  • Reviewer a3zu raised an additional question regarding the comparisons on ZJU-MoCap. We provided further clarification but have not yet received additional feedback.
  • Reviewer v7V1 has not provided substantive feedback on our additional experiments and clarifications.

We believe that the substantial additional experiments and technical clarifications have addressed the reviewers’ major concerns. We kindly ask the Area Chair to consider these revisions and our detailed responses in the final assessment.

Thank you for your time and consideration.

Best regards,
The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.