acceptodds
Under review as a conference paper at ICLR 2027

AHA! Animating Human Avatars and Dynamic Objects in 3D Scenes with Gaussian Splatting

Abstract

We present AHA!, a framework that animates human avatars navigating and interacting with dynamic objects in 3D scenes, with 3D Gaussian Splatting (3DGS) as the only 3D representation for rendering. Existing animation pipelines rely on meshes or point clouds for scenes and humans. These provide the surfaces, distance fields, and walkable regions that motion synthesis and contact modeling need, but they do not provide photorealistic renderings and are hard to recover from casually captured video. 3DGS offers photorealism and reconstruction from a single mobile-phone video, yet provides none of these geometric quantities. We close this gap with four components that operate directly on opacity-filtered Gaussians: occupancy maps for path planning, an egocentric walkability grid that lets a locomotion policy trained only in synthetic mesh rooms run zero-shot in 3DGS scenes, a differentiable SDF proxy for synthesizing scene interactions, and a differentiable contact refinement for human-scene, human-object, and object-scene contact. Because maps and contact targets are functions of the current object poses, the same stack composes long animations in which an avatar walks through a scene, sits down, lifts objects, carries them, and puts them down, re-planning as its own actions change the layout. On SuperSplat and DL3DV scenes, human and VLM preference studies favor our renderings over the strongest mesh pipelines, and our motion quality is within 5% of a clean-mesh upper bound. Full-resolution videos are available at https://anonymous.4open.science/w/AHAICLR27/.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Neural Rendering, Human Animation, Virtual Reality, 3D Humans, 3D Gaussian Splatting, Human-Scene Interaction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_eEztOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents AHA!, a system for animating Gaussian humans in reconstructed 3D Gaussian Splatting (3DGS) scenes. Its central approach is to derive navigation and interaction cues directly from Gaussians rather than extracting scene and object meshes. These cues include occupancy maps, local walkability grids, and approximate distance queries. The system integrates these representations with existing motion-generation components, captured human–object interaction clips, contact refinement, and map updates after object relocation.The evaluation compares motion quality and task completion against mesh-based pipelines, assesses visual preference through user studies, and includes ablations of the navigation, contact, and transition components.

Critical strengths
  1. A practically relevant integration of reconstruction and animation. The paper addresses a useful gap between visually compelling 3DGS reconstructions and the geometric information required for character navigation and interaction. Supporting these capabilities without explicitly extracting scene and object meshes could make reconstructed environments easier to use for animation. The integrated demonstrations provide evidence that the proposed interfaces are workable beyond an isolated navigation task.

  2. Component-level validation of the system design. The ablations examine several consequential components, including Gaussian filtering, local obstacle awareness, dynamic map updates, and contact refinement. These experiments help establish that the proposed system does not rely solely on appealing renderings and that its components contribute to motion or interaction quality.

Critical weaknesses

1.The practical advantage over a mesh-based hybrid pipeline is not yet sufficiently established. The paper demonstrates the feasibility of direct Gaussian-based navigation and interaction, with improvements on some metrics. However, retaining Gaussian rendering while using meshes for geometric queries remains a compelling alternative, given the mature collision-detection and physics tools available in existing pipelines. Avoiding mesh extraction alone does not establish a clear practical advantage. The paper provides limited evidence on the trade-offs in geometric reliability, preprocessing and manual cleanup, runtime cost, and compatibility with existing tools.

2.Limited motion diversity and interaction quality. While the supplementary video shows a range of scenes and characters, the demonstrated motions and interaction remain relatively limited. Several sequences also exhibit unnatural arm poses and hand movements.

Questions for authors
  1. The proposed approach and the baselines use different motion-generation methods. Could the authors explain this choice, or provide a comparison using the same motion-generation method to better isolate the contribution of the proposed geometry-processing approach?
  2. Could the authors further validate the reliability of collision detection based on the spherical approximation of Gaussians? For example, a point inside a closed object but far from the Gaussian centers on its surface may still receive a positive distance estimate, potentially resulting in a missed penetration.
Reviewer_FhDoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AHA, a framework that generates human–scene interaction animations using only 3DGS, without any mesh representation. The authors synthesize motion on the SMPL body model and integrate the Gaussian representations only at the rendering stage. To this end, the framework introduces an occupancy map derived from opacity-filtered Gaussians, a walkability grid that enables zero-shot transfer of a mesh-trained RL locomotion policy to 3DGS, a differentiable SDF proxy, and a contact refinement method based on per-Gaussian offsets. Interactions with dynamic objects are handled by retrieving relevant clips from captured data. Experimental results show that AHA substantially reduces artifacts such as skating and penetration compared with mesh-based baselines, while also achieving superior rendering quality.

Critical strengths

S1. Interaction animation in a purely 3DGS environment

According to the authors, this is the first attempt to perform tasks ranging from navigation to object interaction while keeping the scene, humans, and objects entirely as 3D Gaussians, without extracting meshes. Although the work is primarily a system integration, it is meaningful in that it identifies and addresses the challenges arising at each stage from the lack of explicit geometry in Gaussian representations.

S2. Experimental justification

Mesh-based baselines that use meshes extracted from Gaussians exhibit approximately 3$\times$ more skating and 5–6$\times$ more penetration. This provides a compelling justification for querying geometry directly from the Gaussian representation.

Critical weaknesses

W1. Limited automation and generalization

The metric scale is estimated manually, and the scripts (pℓp_\ell, aℓa_\ell) are also provided by the user. Object interaction relies on retrieval via label lookup, and the action set is limited to only six types. Consequently, the scope of the claim of "animating dynamic objects" is narrow, which should be clarified in the paper.

W2. Heuristic design of contact handling

Since contact refinement shifts only the contact Gaussians rather than the pose itself, it may cause shape distortion. Furthermore, penetrations outside the contact regions remain uncorrected. In addition, because contact detection relies on vertical velocity and acceleration, a foot sliding along the ground could be incorrectly classified as in contact.

Questions for authors

Q1. Runtime breakdown

Could you provide a runtime breakdown of the entire pipeline? This would help clarify the time budget and identify bottlenecks.

Q2. Analysis of evaluation inputs

How many scenes and sequences were used for evaluation? What are the typical distances between pℓp_\ell and the values of LL? Does performance vary with these settings?

Q3. Pose refinement

How does the system handle cases where the initial pose causes significant penetration outside the contact regions? Is refining only the Gaussians while keeping the poses fixed sufficient?

Q4. Generalization of motion retrieval

Can the retrieval process be automated beyond a simple label lookup?

Reviewer_RvQROpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents AHA!, a framework for animating human avatars and dynamic objects in 3D scenes using Gaussian Splatting as the rendering representation. The proposed framework constructs occupancy maps from opacity-filtered Gaussians for path planning, derives egocentric walkability grids to transfer a mesh-trained locomotion policy to Gaussian scenes, and introduces a differentiable distance proxy and contact refinement for human–scene and human–object interactions. Dynamic object interactions are handled through motion retrieval, inbetweening, and occupancy map updates.The framework is evaluated on ScanNet++, SuperSplat, and DL3DV, with comparisons against mesh-based pipelines, rendering preference studies, and component ablations. The reported results demonstrate improved rendering preferences and competitive motion quality, while enabling animations in scenes reconstructed from casual video captures.

Critical strengths

1. A coherent framework for animation in Gaussian scenes. The paper addresses an important practical challenge in 3DGS-based animation by integrating navigation, locomotion, interaction synthesis, and contact refinement into a unified framework. The ability to animate both humans and dynamic objects without extracting explicit meshes from scenes or objects is a meaningful contribution.

2. Promising zero-shot transfer and comprehensive demonstrations. Transferring a locomotion policy trained in synthetic mesh environments to Gaussian scenes without retraining is an interesting aspect of the approach. The evaluations across different scene types, together with the component ablations and qualitative demonstrations, provide encouraging evidence of the framework's practical applicability.

Critical weaknesses

1. Insufficient justification for the Gaussian-only representation.

The paper overlooks existing hybrid mesh–Gaussian methods [1–3] that combine explicit geometry with photorealistic rendering. Although these methods address different tasks, they challenge the assumed trade-off between geometric information and rendering quality. The advantages of AHA! over hybrid approaches remain insufficiently justified.

2. Confounded motion-quality comparisons.

In Table 1, Baselines A and B use DIMOS, whereas AHA! employs a different motion generation pipeline (Appendix C, Table 9). Thus, the reported improvements cannot be attributed solely to the Gaussian representation. A controlled comparison using the same motion model and policy, varying only the geometric interface, is needed.

3. Insufficient independent validation of contact accuracy.

The contact-distance metric closely matches the Gaussian-center distance minimized by Eq. (7). Although additional metrics are reported, they do not fully establish accurate surface contact. In particular, nearest-center approximations may not capture anisotropic Gaussian geometry. Independent evaluation against reference surface geometry would strengthen the results.

References

[1] Shao, Z., et al. SplattingAvatar: Realistic Real-Time Human Avatars with Mesh-Embedded Gaussian Splatting. CVPR, 2024.

[2] Choi, et al. MeshGS: Adaptive Mesh-Aligned Gaussian Splatting for High-Quality Rendering. ACCV, 2024.

[3] Luo, et al. Tavatar: Topology-Aware Gaussian Attribute Derivation for Animatable Human Avatars. CVPR, 2026.

Questions for authors

Q1. Gaussian-only versus hybrid representations. What are the specific advantages of the proposed Gaussian-only scene representation over a hybrid mesh–Gaussian approach that uses meshes for geometric queries while retaining Gaussians for rendering? Can the authors provide quantitative evidence regarding reconstruction robustness, computational efficiency, or motion quality to support this design choice?

Q2. Controlled motion-quality comparison. Could the authors report quantitative results for the additional mesh-derived walkability grid experiment mentioned in Appendix C? In particular, how does the method perform when the motion generator, locomotion policy, and training conditions are kept identical, and only the geometric representation used for navigation and collision handling is changed?

Q3. Independent contact evaluation. Could the authors evaluate contact accuracy against an independent geometric reference, such as ground-truth mesh surfaces where available? This would help clarify whether the improvements in Gaussian-center contact distance translate into more accurate surface contact and reduced geometric penetration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.