- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces a method for generating 4D motion sequences given the 3D shape of an object and a driving video. It also supports 3D representations that incorporate a skeleton, enabling the direct generation of skeletal pose sequences. The presented results indicate that the proposed method performs well, outperforming baseline approaches.
Critical strengths
- Unlike previous approaches, this paper does not learn a motion latent space; instead, it directly learns a vertex trajectory field to perform denoising at the level of individual points in 3D space, offering a novel solution.
- The proposed method enables both 4D mesh reconstruction and skeletal pose estimation from a given video.
- The technical approach appears concise, and experimental results demonstrate that it outperforms existing methods.
- Capable of handling video sequences of arbitrary length.
Critical weaknesses
- Since the proposed method learns vertex trajectories directly, instead of relying on pre-constructed motion latents, it may be more sensitive to mesh resolution, topological connectivity, and even minor errors introduced during denoising. This is particularly relevant for non-uniform meshes, where regions with finer details (such as fingers) feature a higher density of vertices, while other areas (such as the torso) have a sparser distribution. The paper should include an ablation study that keeps the 3D shape and input pose constant while varying parameters such as mesh resolution, topological connectivity, and vertex density.
- Although the proposed method achieves superior fitting quality, some results exhibit some noticeable jitter, as seen in the accompanying videos (specifically the animations between the 17-second and 26-second marks).
- Can the proposed method effectively handle self-occlusion?
- Learning global object rotation using a vertex-trajectory-based representation is more challenging than with rotation-related representations; how well does the proposed method perform when the object undergoes its own rotation?
- How effectively does the proposed method handle fine-grained details, such as the motion of individual fingers?
Questions for authors
The proposed method does not discuss its scope of applicability or its limitations; therefore, I have concerns regarding the following points:
- the impact of mesh quality;
- self-occlusion in monocular video;
- significant rotational motion of an object;
- fine-grained deformation details.
2 comments
Official Comment by Reviewer C428
Thank you for the detailed responses, additional experiments, and visual results, which have addressed some of my concerns and helped me better understand the scope and limitations of the proposed method.
One minor limitation of the object rotation experiment is that it appears to involve only rigid rotations, without any non-rigid motion. Incorporating motion, such as a character walking while turning around, would better reflect scenarios commonly encountered in real-world input videos.
Nevertheless, I believe the method demonstrates strong performance and offers novel insights. Therefore, I will maintain my original rating.
We thank the reviewer for recognizing the direct trajectory formulation and its applicability to both mesh and skeletal tracking. We will expand the analysis of mesh quality and clarify the model's capabilities and limitations in monocular motion recovery.
W1. Mesh resolution, connectivity, and non-uniform vertex density.
FoMo is robust to different mesh resolutions of the same object. Quantitative evaluation of this property is challenging because the available datasets do not provide meshes of the same object at multiple resolutions. To examine robustness across mesh densities, we fix the driving video and use Tripo P2 to reconstruct an orange cat at three resolutions: 2k, 4k, and 8k vertices. We show the corresponding 3D animations from multiple views below (in the anonymous link):
Density Comparison Video
W2. Temporal jitter.
Temporal accuracy is already part of our quantitative evaluation: the vertex velocity error (VE) in Table 1 measures errors in corresponding-vertex displacements between adjacent frames, including those introduced by frame-to-frame jitter. FoMo achieves the lowest VE among the evaluated methods in both the full-sequence and 16-frame settings, under both reference protocols. Thus, the quantitative results already account for temporal motion errors and show an advantage over the baselines, alongside the improvements in positional accuracy.
The examples identified by the reviewer show that residual jitter can remain even with this improvement. We agree that further reducing it would improve animation quality. The current model has approximately 186.6M trainable parameters and is trained on roughly 16k animation sequences; increasing model capacity and motion coverage offers a promising direction for further improvement.
W3. Self-occlusion.
FoMo predicts complete trajectories, including points that are invisible in individual frames. The reference geometry provides object-structure information, temporal attention connects observations across frames, and the learned conditional motion prior helps infer motion when direct image evidence is unavailable. Together, these mechanisms support reasoning about self-occlusion. In the examples below (in the anonymous link), even when the input contains only frontal views of animals such as deer and bears, FoMo produces natural motion for their unseen rear regions.
Self-occlusion example video
W4. Significant object rotation.
Despite modeling motion at the level of individual vertices, FoMo's generated trajectories effectively capture global rotation. The examples below show FoMo's reconstructions from videos of rotating objects and characters.
rotation example video
W5. Fine-grained deformation.
Modeling finger animation in isolation is feasible within our vertex-trajectory formulation. In the current full-body setting, however, all input frames are resized to 256 × 256, where hands often occupy too few pixels for individual finger movements to be clearly visible. This limits the visual evidence available to recover fine-grained hand motion. Reliable recovery of such motion is therefore a limitation at the current input resolution and training scale. Higher-resolution or hand-focused inputs, together with richer fine-grained motion data, are natural directions for improvement.