acceptodds
Under review as a conference paper at ICLR 2027

HoAM-4D: Holistic Aware Mapping for 4D Mesh Animation and Reconstruction

Abstract

While generative 3D modeling has advanced rapidly, lifting static 3D assets and monocular videos into coherent 4D mesh sequences remains challenging, as end-to-end feed-forward frameworks often lack explicit video–mesh alignment, while view-dependent optimization methods struggle to model authentic 4D motion, resulting in inconsistent image observations or disordered trajectories. To address these challenges, we present HoAM-4D, a multi-stage framework that introduces holistic vertex awareness through 4D kinematic flow and video-to-mesh mapping for coherent 4D mesh generation. Specifically, to improve 4D motion modeling, HoAM-4D projects both the 3D mesh and video observations into 3D space and predicts geometry-motion and image-motion displacement fields. To address video–mesh misalignment, it further predicts image-to-canonical-mesh and image-to-synchronous-mesh correspondence fields. Beyond visible regions, we extend the video-to-mesh mapping to occluded areas through skeleton-based conduction. Finally, we leverage both the input video and 4D motion as supervision to optimize holistic-aware mesh deformation from the first-frame canonical mesh, enforcing articulated structural, temporal, and appearance consistency. Extensive experiments demonstrate that HoAM-4D outperforms existing methods in geometric accuracy, motion coherence, and video-aligned rendering quality.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Monocular 4D Reconstruction, Animatable Mesh Reconstruction, Holistic Video mesh awarness

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_cgHcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Taking a monocular video, this paper produces a mesh that is ready for animation. The method uses the Cross-Modal Motion Estimation Transformer that lifts the video into a mesh and motion, across 4 fields: geometry motion, image motion, and image-to-canonical and image-to-synchronous correspondence offsets. Then the method uses Skeleton-based Deviation Conduction to improve occluded region completeness and refine temporal coherence. The paper compares against relevant methods on ActionBench and some high-resolution video-only sequences.

Critical strengths
  • The modules in the network are clear, novel, and the ablation section supports the various sections of the method clearly.
  • The paper is well validated across datasets against many relevant baselines.
Critical weaknesses
  • Some method details are unclear in the main text, such as the vertex correspondence and depth of image pixels.
  • The reliability of initialization with Hunyuan models is large undiscussed or validated. The domain shift from GT training data to the generated meshes is not discussed in detail.
  • The runtime of this method is much slower than prior methods.
Questions for authors
  • Does the depth of image features come from depth estimation, or the Hunyuan3D generated mesh?
  • How is the across-frame vertex correspondence obtained for the generated meshes? How noisy are these correspondences? And how does the noise affect the output results?
  • How does inaccurate or differing generation across frames impact the downstream work after scene initialization?
Reviewer_cKknOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces HoAM-4D, a method for reconstructing 4D meshes from monocular videos. It jointly constrains the deformation of an initial mesh through 3D geometric motion, video–mesh correspondence, temporal continuity, and skeletal structure. These constraints enable the reconstructed 4D mesh to faithfully follow the input video while preserving consistent topology, plausible articulated motion, and temporal coherence.

Critical strengths

The unified and coordinated modeling of motion and correspondence is novel. The paper represents geometric motion, image motion, and video–mesh correspondence as 3D displacement fields, allowing predictions from different sources to cross-check one another through cross-field consistency constraints in a shared coordinate system.

Critical weaknesses

The ablation studies of the key constraints are insufficient. Although the paper ablates some supervision terms and network components, the geometric cycle-consistency loss LcycL_{cyc} and motion-consistency loss LmmL_{mm} are removed only jointly, making it difficult to distinguish their individual contributions and complementarity. The paper also lacks separate ablations of the additional displacement prediction heads and simplified baselines that derive some fields from fewer predicted fields, leaving the necessity of jointly modeling all four fields insufficiently established. Furthermore, skeleton-based propagation to occluded regions is not evaluated separately while retaining supervision for visible regions.

Questions for authors

Could the authors provide additional quantitative and qualitative ablation studies addressing the concerns raised in the Weaknesses section?

2 comments

We thank the reviewer for recognizing the novelty of modeling geometric motion, image motion, and video–mesh correspondence as 3D displacement fields that cross-check one another in a shared coordinate system. The requested ablations now quantify this design directly:

  • Both consistency losses matter. The geometric-cycle loss Lcyc\mathcal{L} _{cyc} and the matched-motion loss Lmm\mathcal{L} _{mm} each account for about half of the joint gain, and they act on different fields (Q1).
  • All four fields are necessary. Every reduced set falls short of the full model, by up to 10.7 Acc (Q2).
  • Skeleton conduction reaches occluded parts. It lowers CD-4D from 0.068 to 0.062 and CD-M from 0.127 to 0.118. The gain is largest on vertices invisible from the camera, where CD_occ (CD-4D restricted to these vertices) drops from 0.176 to 0.134 (Q3).

All changes are in blue in the revision; table and section numbers refer to the revised version.

Q1: Individual effect of Lcyc\mathcal{L} _{cyc} and Lmm\mathcal{L} _{mm}. Added to Table 4:

VariantAcc↑CLIP↑CD-4D↓CD-M↓
Full model81.590.93120.0620.118
w/o Lcyc\mathcal{L} _{cyc} + Lmm\mathcal{L} _{mm}79.040.91530.0670.135
w/o Lcyc\mathcal{L} _{cyc} (new)79.870.92480.0640.127
w/o Lmm\mathcal{L} _{mm} (new)80.930.92210.0650.126

Without Lcyc\mathcal{L} _{cyc}, the correspondence offsets are no longer tied to the geometry motion, so Acc drops the most. Without Lmm\mathcal{L} _{mm}, the image motion is no longer tied to the transferred geometry motion, so CLIP and CD-4D drop more. Each removal costs about half of the joint CD-M increase (+0.009 and +0.008 vs. +0.017), so the two terms are complementary.

Q2: Necessity of all four displacement fields. DG\mathbf{D}^G is the anchor. Without it, Stage III can only propagate 2D image motion back to the mesh, which is the existing "w/o motion sup." row of Table 3 (CD-4D 0.062 → 0.142). We keep DG\mathbf{D}^G and ablate the other fields in new Appendix D.2 (Table 8). A removed field is derived from the rest via Osync=Ocan+DG\mathbf{O}^{sync}=\mathbf{O}^{can}+\mathbf{D}^G, so Stage III always receives an image-to-canonical correspondence. Some variants are evaluated with existing checkpoints from earlier development that lack the removed heads; the others are fine-tuned from the 70%-epoch checkpoint with the corresponding heads and objective. All variants will be fully retrained from scratch in the final version. The results are:

Fields predictedDerivedAcc↑CLIP↑CD-4D↓CD-M↓CD_occ↓
All four (full)–81.590.93120.0620.1180.134
w/o Osync\mathbf{O}^{sync}Osync\mathbf{O}^{sync}80.030.92550.0640.1260.142
w/o DI\mathbf{D}^{I}DI\mathbf{D}^{I}78.690.90500.0670.1300.148
w/o Ocan\mathbf{O}^{can}Ocan\mathbf{O}^{can}78.460.91960.0670.1300.151
(i) DG\mathbf{D}^G+Ocan\mathbf{O}^{can}Osync\mathbf{O}^{sync}75.520.90340.0700.1360.160
(ii) DG\mathbf{D}^G+Osync\mathbf{O}^{sync}Ocan\mathbf{O}^{can}72.940.89980.0730.1390.171
(iii) DG\mathbf{D}^G+DI\mathbf{D}^{I}Ocan\mathbf{O}^{can}70.850.91080.0740.1390.189

Removing Ocan\mathbf{O}^{can} hurts the most, as the canonical map must be computed as Osync−DG\mathbf{O}^{sync}-\mathbf{D}^G and both fields' errors add up. Removing DI\mathbf{D}^I costs 2.9 Acc. We attribute this to branch-specific supervision. Geometry voxels carry semantic features averaged over multi-view renderings, while image voxels carry clean per-frame features. When DG\mathbf{D}^G and DI\mathbf{D}^I supervise their own branches, each branch keeps its modality-specific features and the cross-modal layers focus on interaction. Without DI\mathbf{D}^I, the image branch is trained only by interaction losses and loses frame-specific detail early. Variant (iii) is the weakest except in CLIP, which DI\mathbf{D}^I keeps high: it recovers the canonical map by nearest-endpoint matching, which is ambiguous along the viewing direction because image voxels have zero-padded depth.

Q3: Effect of skeleton conduction to occluded regions. New row in Table 3: visible-region deviation is kept, but its redistribution to occluded vertices is disabled.

VariantCLIP↑FVD↓CD-4D↓CD-M↓CD_occ↓
Full model0.9312471.280.0620.1180.134
w/o skeleton conduction (new)0.9263489.620.0680.1270.176
w/o 2D deviation sup.0.9124536.850.0890.1310.208

Occluded parts of a rigid limb then lag behind the visible parts and stretch the mesh near the visibility boundary. CD_occ lies between the full model and the variant without 2D supervision, so conduction supplies the occluded-region accuracy that visible supervision alone misses.

We hope these ablations address the reviewer's concerns and welcome further questions.

Reviewer_cKknOpenReview ↗

Thanks for the further results

Thank you for the helpful clarifications and additional results. These have resolved my concerns, and I have increased my score accordingly.

Reviewer_9npJOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

HoAM-4D reconstructs a mesh animation from monocular video by jointly predicting motion and video-to-mesh correspondences, then fitting a canonical mesh using skeletal constraints.

Critical strengths
  • Addresses a useful problem and preserves mesh topology throughout the animation.
  • Combines geometric and image-derived evidence in a plausible framework.
  • Reports improvements across geometry, motion, and rendering metrics, with several architectural ablations.
Critical weaknesses
  • The paper does not explain how 2D image features acquire 3D voxel coordinates, particularly depth. It also leaves unclear how displacement fields defined on current-frame geometry are transferred to canonical-mesh vertices.
  • Unfair ablation. Replacing correspondence regression with classification also removes cycle and matched-motion losses. These losses can be constructed using a classifier’s expected endpoints, so Table 4 does not isolate the advantage of regression.
  • Missing baseline. Some related works are missing, e.g. Motion3to4[1] and Mesh4D[2].

[1] Motion 3-to-4: 3D Motion Reconstruction for 4D Synthesis (CVPR 2026) [2] Mesh4D: 4D Mesh Reconstruction and Tracking from Monocular Video (CVPR 2026)

Questions for authors

Could you please clarify the points in the weakness section?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.