- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Taking a monocular video, this paper produces a mesh that is ready for animation. The method uses the Cross-Modal Motion Estimation Transformer that lifts the video into a mesh and motion, across 4 fields: geometry motion, image motion, and image-to-canonical and image-to-synchronous correspondence offsets. Then the method uses Skeleton-based Deviation Conduction to improve occluded region completeness and refine temporal coherence. The paper compares against relevant methods on ActionBench and some high-resolution video-only sequences.
Critical strengths
- The modules in the network are clear, novel, and the ablation section supports the various sections of the method clearly.
- The paper is well validated across datasets against many relevant baselines.
Critical weaknesses
- Some method details are unclear in the main text, such as the vertex correspondence and depth of image pixels.
- The reliability of initialization with Hunyuan models is large undiscussed or validated. The domain shift from GT training data to the generated meshes is not discussed in detail.
- The runtime of this method is much slower than prior methods.
Questions for authors
- Does the depth of image features come from depth estimation, or the Hunyuan3D generated mesh?
- How is the across-frame vertex correspondence obtained for the generated meshes? How noisy are these correspondences? And how does the noise affect the output results?
- How does inaccurate or differing generation across frames impact the downstream work after scene initialization?
We thank the reviewer for recognizing the novelty of modeling geometric motion, image motion, and video–mesh correspondence as 3D displacement fields that cross-check one another in a shared coordinate system. The requested ablations now quantify this design directly:
All changes are in blue in the revision; table and section numbers refer to the revised version.
Q1: Individual effect of Lcyc and Lmm. Added to Table 4:
Without Lcyc, the correspondence offsets are no longer tied to the geometry motion, so Acc drops the most. Without Lmm, the image motion is no longer tied to the transferred geometry motion, so CLIP and CD-4D drop more. Each removal costs about half of the joint CD-M increase (+0.009 and +0.008 vs. +0.017), so the two terms are complementary.
Q2: Necessity of all four displacement fields. DG is the anchor. Without it, Stage III can only propagate 2D image motion back to the mesh, which is the existing "w/o motion sup." row of Table 3 (CD-4D 0.062 → 0.142). We keep DG and ablate the other fields in new Appendix D.2 (Table 8). A removed field is derived from the rest via Osync=Ocan+DG, so Stage III always receives an image-to-canonical correspondence. Some variants are evaluated with existing checkpoints from earlier development that lack the removed heads; the others are fine-tuned from the 70%-epoch checkpoint with the corresponding heads and objective. All variants will be fully retrained from scratch in the final version. The results are:
Removing Ocan hurts the most, as the canonical map must be computed as Osync−DG and both fields' errors add up. Removing DI costs 2.9 Acc. We attribute this to branch-specific supervision. Geometry voxels carry semantic features averaged over multi-view renderings, while image voxels carry clean per-frame features. When DG and DI supervise their own branches, each branch keeps its modality-specific features and the cross-modal layers focus on interaction. Without DI, the image branch is trained only by interaction losses and loses frame-specific detail early. Variant (iii) is the weakest except in CLIP, which DI keeps high: it recovers the canonical map by nearest-endpoint matching, which is ambiguous along the viewing direction because image voxels have zero-padded depth.
Q3: Effect of skeleton conduction to occluded regions. New row in Table 3: visible-region deviation is kept, but its redistribution to occluded vertices is disabled.
Occluded parts of a rigid limb then lag behind the visible parts and stretch the mesh near the visibility boundary. CD_occ lies between the full model and the variant without 2D supervision, so conduction supplies the occluded-region accuracy that visible supervision alone misses.
We hope these ablations address the reviewer's concerns and welcome further questions.