acceptodds
Under review as a conference paper at ICLR 2027

FoMo: Generative 4D Morphing for Motion Tracking

Abstract

We present FoMo, a generative model for 4D motion tracking of a known mesh from monocular video. We formulate the problem as generative 4D morphing natively in 4D vertex trajectory space with flow matching, with FoMo most notably resolving the resulting noise-to-vertex transport ambiguity through a dual-geometry architecture. FoMo significantly outperforms prior methods on mesh tracking and, when applied to skeletal tracking, also surpasses specialized skeletal trackers. It further generalizes to in-the-wild and synthetic videos, enabling tracking of reconstructed real-world meshes, cross-identity motion capture, and the use of video generation models as 4D motion generators.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
4D Motion, 4D Generation, Animation, Mesh, Graphics.

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_C428OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a method for generating 4D motion sequences given the 3D shape of an object and a driving video. It also supports 3D representations that incorporate a skeleton, enabling the direct generation of skeletal pose sequences. The presented results indicate that the proposed method performs well, outperforming baseline approaches.

Critical strengths
  • Unlike previous approaches, this paper does not learn a motion latent space; instead, it directly learns a vertex trajectory field to perform denoising at the level of individual points in 3D space, offering a novel solution.
  • The proposed method enables both 4D mesh reconstruction and skeletal pose estimation from a given video.
  • The technical approach appears concise, and experimental results demonstrate that it outperforms existing methods.
  • Capable of handling video sequences of arbitrary length.
Critical weaknesses
  • Since the proposed method learns vertex trajectories directly, instead of relying on pre-constructed motion latents, it may be more sensitive to mesh resolution, topological connectivity, and even minor errors introduced during denoising. This is particularly relevant for non-uniform meshes, where regions with finer details (such as fingers) feature a higher density of vertices, while other areas (such as the torso) have a sparser distribution. The paper should include an ablation study that keeps the 3D shape and input pose constant while varying parameters such as mesh resolution, topological connectivity, and vertex density.
  • Although the proposed method achieves superior fitting quality, some results exhibit some noticeable jitter, as seen in the accompanying videos (specifically the animations between the 17-second and 26-second marks).
  • Can the proposed method effectively handle self-occlusion?
  • Learning global object rotation using a vertex-trajectory-based representation is more challenging than with rotation-related representations; how well does the proposed method perform when the object undergoes its own rotation?
  • How effectively does the proposed method handle fine-grained details, such as the motion of individual fingers?
Questions for authors

The proposed method does not discuss its scope of applicability or its limitations; therefore, I have concerns regarding the following points:

  • the impact of mesh quality;
  • self-occlusion in monocular video;
  • significant rotational motion of an object;
  • fine-grained deformation details.
2 comments

We thank the reviewer for recognizing the direct trajectory formulation and its applicability to both mesh and skeletal tracking. We will expand the analysis of mesh quality and clarify the model's capabilities and limitations in monocular motion recovery.

W1. Mesh resolution, connectivity, and non-uniform vertex density.

FoMo is robust to different mesh resolutions of the same object. Quantitative evaluation of this property is challenging because the available datasets do not provide meshes of the same object at multiple resolutions. To examine robustness across mesh densities, we fix the driving video and use Tripo P2 to reconstruct an orange cat at three resolutions: 2k, 4k, and 8k vertices. We show the corresponding 3D animations from multiple views below (in the anonymous link):

Density Comparison Video

W2. Temporal jitter.

Temporal accuracy is already part of our quantitative evaluation: the vertex velocity error (VE) in Table 1 measures errors in corresponding-vertex displacements between adjacent frames, including those introduced by frame-to-frame jitter. FoMo achieves the lowest VE among the evaluated methods in both the full-sequence and 16-frame settings, under both reference protocols. Thus, the quantitative results already account for temporal motion errors and show an advantage over the baselines, alongside the improvements in positional accuracy.

The examples identified by the reviewer show that residual jitter can remain even with this improvement. We agree that further reducing it would improve animation quality. The current model has approximately 186.6M trainable parameters and is trained on roughly 16k animation sequences; increasing model capacity and motion coverage offers a promising direction for further improvement.

W3. Self-occlusion.

FoMo predicts complete trajectories, including points that are invisible in individual frames. The reference geometry provides object-structure information, temporal attention connects observations across frames, and the learned conditional motion prior helps infer motion when direct image evidence is unavailable. Together, these mechanisms support reasoning about self-occlusion. In the examples below (in the anonymous link), even when the input contains only frontal views of animals such as deer and bears, FoMo produces natural motion for their unseen rear regions.

Self-occlusion example video

W4. Significant object rotation.

Despite modeling motion at the level of individual vertices, FoMo's generated trajectories effectively capture global rotation. The examples below show FoMo's reconstructions from videos of rotating objects and characters.

rotation example video

W5. Fine-grained deformation.

Modeling finger animation in isolation is feasible within our vertex-trajectory formulation. In the current full-body setting, however, all input frames are resized to 256 × 256, where hands often occupy too few pixels for individual finger movements to be clearly visible. This limits the visual evidence available to recover fine-grained hand motion. Reliable recovery of such motion is therefore a limitation at the current input resolution and training scale. Higher-resolution or hand-focused inputs, together with richer fine-grained motion data, are natural directions for improvement.

Reviewer_C428OpenReview ↗

Official Comment by Reviewer C428

Thank you for the detailed responses, additional experiments, and visual results, which have addressed some of my concerns and helped me better understand the scope and limitations of the proposed method.

One minor limitation of the object rotation experiment is that it appears to involve only rigid rotations, without any non-rigid motion. Incorporating motion, such as a character walking while turning around, would better reflect scenarios commonly encountered in real-world input videos.

Nevertheless, I believe the method demonstrates strong performance and offers novel insights. Therefore, I will maintain my original rating.

Reviewer_Yn5TOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a generative method to reconstruct motions of a given mesh conditioned on a monocular video using flow matching. For this, it introduces a “dual-geometry transformer” which embed tokens under a reference block and a dynamic generation block.

Critical strengths

Problem formulation and motivations are clear and make sense to me.

The paper is well-written and easy to follow even though I would have preferred equation formulations of the dual geometry transformer. If possible, I suggest adding those.

The provided quantitative and qualitative comparisons are convincing to support the superiority of FoMo over its competitors

Critical weaknesses

My main concern is about the evaluation protocol: FoMo is a generative model but it compares deterministic trajectory synthesis on only one seed (as far as I see in the paper). The learned distribution of motions is not evaluated (with FID or CLIP score on rendered frames for instance).

Second is the ablation study: removing reference attention is something but what about using simpler alternatives to attention (raw coordinates or a MLP)? Also, I would be curious to see how sensible the architecture is to the image encoder (what if you swap DINOV3 to another?)

Then, about the experimental evaluation, I only see one evaluation dataset for which the method is trained on. In-the-wild applications seem good but quantitative evaluation with metrics on large scale test sets would make this more convincing.

Finally, you claim robustness to mesh resolution which is not rigorously evaluated. Test meshes have up to 13,000 vertices which is far from scan resolution (up to millions of vertices with intricate details)

Questions for authors
  1. I understand the method is quite expensive to train but do you think it would perform well if trained and/or tested on DeformingThings4D for instance (or any other such dataset) ?

  2. Have you tried with very high resolution meshes ? And low resolution videos ?

  3. How expensive is the inference in terms of GPU usage and time ?

  4. How much seed-sensitive is the method ?

4 comments

We thank the reviewer for the constructive feedback and for finding the formulation clear and the quantitative and qualitative comparisons convincing. We address visual evaluation, seed sensitivity, ablations, generalization, resolution, and inference cost below.

W1: Visual evaluation of the generated motion.

We agree that the learned distribution of motions should be evaluated with FID or CLIP score on rendered frames for instance.We have supplemented the evaluation with the visual metrics on Motion-80, for both full sequences and the first 16 frames. Evaluation uses renderings from four orthogonal views. CLIP measures image-feature similarity between predicted and ground-truth renderings, while FID is computed by pooling rendered frames from all four views.

MethodFull-sequence CLIP ↑Full-sequence FID ↓First-16-frame CLIP ↑First-16-frame FID ↓
FoMo0.9702288.5659250.97094511.487064
Motion 3-to-40.96087910.6571960.96332413.198238
R-DMesh0.94247119.3458700.95298316.465168
ActionMesh0.92931133.2937710.96359213.001058

FoMo achieves the highest CLIP similarity and lowest FID among the evaluated methods in both sequence-length settings. These results complement the correspondence-aware tracking metrics with evidence of improved rendered visual quality.

W2: Simpler reference conditioning.

We have added a parameter-matched simple-encoding alternative on Motion-80 (16 frames), using the same standard 120k-step training setup as the ablations in Table 4. This alternative encodes the reference mesh coordinates, RGB colors, and surface normals, then directly concatenates the resulting reference features with the noise input without using the dual-geometry architecture. Each DiT block then consists of spatial attention, temporal attention, and image cross-attention. Its parameter count is matched to FoMo for a fair comparison.

We evaluate both methods under both reference protocols:

ProtocolModelCD ↓VE ↓L1 ↓L2 ↓
w/ PASimple-encoding alternative0.0760.0290.1020.072
w/ PAFoMo0.0670.0240.0870.062
w/o PASimple-encoding alternative0.1280.0330.2270.159
w/o PAFoMo0.1210.0300.1950.139

FoMo outperforms the simple-encoding alternative on all four metrics under both protocols. The advantage supports the benefit of the dual-geometry design over input concatenation alone under matched model capacity and training budget. We have updated Section 4.5.2 accordingly, with changes highlighted in blue. Please see the anonymous video below for qualitative comparisons.

Qualitative Comparison

W2: Robustness for vision encoder. To examine whether our performance depends on the choice of visual backbone, we replace the frozen DINOv3 ViT-H/16+ encoder with a frozen DINOv2 ViT-L/14 encoder with registers, adapting the image-feature projections accordingly. We retain the same model architecture and training settings, train the Dinov2 variant for 120k steps and evaluate on Motion-80 (16 frames) as above. We report the results:

Vision encoderCD ↓VE ↓L1 ↓L2 ↓
DINOv30.0670.0240.0870.062
DINOv20.0680.0240.0860.061

The difference is tiny. This comparison supports the robustness of our approach to the visual backbone choice and indicates that the performance is not specific to DINOv3.

W1, Q4: Seed sensitivity.

We independently sampled and ran inference with three random seeds on Motion-80 (16 frames), using a fixed model checkpoint. The mean and sample standard deviation across the three benchmark-level results are:

SeedCD ↓VE ↓L1 ↓L2 ↓
seed_10.049796040.016765390.055611390.03968485
seed_20.050117320.016846250.056341170.04012027
seed_30.050216930.016754630.055956490.04000682
Mean0.050043430.016788760.055969680.03993731
Sample standard deviation0.000219960.000050080.000365070.00022588

The small variation across these independent inference runs indicates stable aggregate tracking performance with respect to the sampling seed on this benchmark.

W4: High-resolution meshes, low-resolution videos, and long sequences.

For video resolution, FoMo is trained on 256 × 256 frames, and all input videos are automatically resized to 256 × 256 at inference. The reported results therefore already use relatively low-resolution visual input.

For mesh resolution, many practical animation assets, including action-game characters, use meshes with thousands to tens of thousands of vertices. More fundamentally, surface resolution and the intrinsic degrees of freedom of motion are distinct. We view video-to-vertex-trajectory prediction as a natural end-to-end learning objective because it directly supervises the desired motion with explicit correspondence. This objective does not require treating every surface sample as an independent motion degree of freedom. For very dense assets, a sufficiently informative set of point trajectories can be predicted and propagated to the dense surface through subdivision or geodesic-aware, locally rigid skinning. The principle is to resolve the underlying deformation at an appropriate sampling density, then evaluate it at the desired surface resolution. This provides a route to animating million-vertex assets while concentrating motion prediction on the information needed to drive them. The required sampling density increases when the surface contains additional independently moving details.

For sequence length, as reported in Appendix B, FoMo can process animation sequences with up to 628 frames. For sequences combining very high vertex counts with longer durations, autoregressive generation is a possible extension.

Q3: Inference GPU usage and runtime.

We report runtime per Euler step and peak allocated GPU memory on a single NVIDIA A800.

Fixed 5,000 vertices; varying target-frame count.

Target framesTime per Euler step (ms)Peak allocated memory (GiB)
4521.5532.998
161,649.9814.362
324,074.8516.350
487,563.8278.555
6412,186.97410.760

Fixed 16 target frames; varying vertex count.

VerticesTime per Euler step (ms)Peak allocated memory (GiB)
2,500664.4673.224
5,0001,652.9874.362
7,5002,935.4915.565
10,0004,572.8706.767

We have added detailed inference-cost measurements to Appendix E and highlighted the additions in blue. Our implementation uses the full 50-step Euler sampler, without sampling acceleration or distillation. Reducing the number of sampling steps and distilling the model offer substantial scope for faster inference. Our focus in this work is direct trajectory modeling and tracking quality; we leave these optimizations to future work.

W3, Q1: Other 4D datasets and broader evaluation.

FoMo can be trained on other 4D mesh datasets, including DeformingThings4D, and does not require a fixed category-specific mesh template. As stated in Section 4.1, Motion-80 uses held-out assets with no model overlap with the training set. Given the size of the DeformingThings4D test set and the cost of evaluating every baseline on it, completing this comparison within the rebuttal period is challenging.

S2: adding equation formulations.

We also appreciate the suggestion to add equations. We have added mathematical descriptions of the reference and generation blocks to 3.3 section, explicitly distinguishing RoPE based on clean reference coordinates from RoPE based on noisy coordinates, and specifying the topology, temporal-attention, and image-conditioning operations. These additions are highlighted in blue in the revised manuscript.

Hi, Thanks for these answers. most of my comments are addressed.

I wonder why the rendering metrics (FID and CLIP) are not reported in the revised paper, I would say these results are a good argument supporting the proposed method

I am still on the edge between recommending acceptance or rejection for two reasons: first the method is evaluated on a single dataset. Of course, evaluating on DT4D is too much for the rebuttal periode, I understand that but then I can only hope the method is data-efficient and generalizable enough to work in various training protocols. Second, the "scalability" of the method seems like a strong limitation: here you only evaluated with meshes up to 10,000 vertices with 16 frames which is still far from some data I am used to work with. But then, I look at the supplementary video and I see a clear superiority of FoMo. These examples could be cherry picked but I am willing to give the benefit of the doubt

That said, I acknowledge the empirical work of this paper. I will raise my score and hope to dig more at the conference and try out the code once released

We thank the reviewer for the support and suggestion. We have updated the revised manuscript to include rendering-based evaluation with CLIP and FID, together with the additional LPIPS and FVD metrics in Table 1. FoMo achieves the best performance on all four metrics across all evaluation protocols. We will also release our code and checkpoint soon.

P.S. “First-16-frame" crops full-sequence predictions, whereas “Motion-80 (16 frames)”in table 1 uses independent 16-frame inputs, yielding slightly different scores. Both settings exclude the effects of autoregressive rollouts.

Reviewer_3YfLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FoMo, a flow-matching model that directly generates indexed point trajectories for monocular mesh tracking, without a learned motion latent. Its dual-geometry transformer combines fixed reference geometry with evolving noisy geometry and handles mesh vertices and skeletal joints within the same representation.

Critical strengths
  • FoMo consistently outperforms the evaluated baselines across all four mesh-tracking metrics and both reference protocols (Table 1). The gains persist in the 16-frame setting, where all methods avoid autoregressive rollout, strengthening the empirical comparison.
  • Direct trajectory modeling provides a useful unified formulation for meshes and skeletons. Table 2 supports its practical value: FoMo performs well through both direct joint prediction and the same mesh-to-skeleton inverse fitting used for competing mesh methods.
Critical weaknesses
  • The ablations support the proposed components, but remove attention modules or reference information rather than comparing against a conditioning- and capacity-matched alternative. It remains unclear how much dual geometry improves over simpler reference conditioning
  • Full-sequence inference at native mesh resolution is demonstrated, but representative runtime and peak-memory measurements are missing. Without these, the quality–computation trade-off of avoiding motion compression remains unclear, particularly for long sequences and dense meshes.
Questions for authors
  • How does FoMo compare with per-token reference-coordinate embeddings and only noisy-coordinate spatial RoPE, keeping reference inputs, attention depth, parameter count, and training budget comparable? Results under both reference protocols using the existing 120k-step setup would help isolate the dual-geometry contribution.
  • Does FoMo’s direct joint prediction in Table 2 use only the reference skeleton, or also the mesh, its attributes, and reference rendering? What reference information is provided to MoCapAnything V2?
2 comments

We thank the reviewer for recognizing the consistent improvements across the mesh-tracking protocols and the value of a unified trajectory representation for meshes and skeletons. We address the controlled comparison, computational cost, and reference inputs below.

W1, Q1: Dual geometry versus parameter-matched simple reference encoding.

We have added a parameter-matched simple-encoding alternative on Motion-80 (16 frames), using the same standard 120k-step training setup as the ablations in Table 4. This alternative encodes the reference mesh coordinates, RGB colors, and surface normals, then directly concatenates the resulting reference features with the noise input without using the dual-geometry architecture. Each DiT block then consists of spatial attention, temporal attention, and image cross-attention. Its parameter count is matched to FoMo for a fair comparison.

We evaluate both methods under both reference protocols:

ProtocolModelCD ↓VE ↓L1 ↓L2 ↓
w/ PASimple-encoding alternative0.0760.0290.1020.072
w/ PAFoMo0.0670.0240.0870.062
w/o PASimple-encoding alternative0.1280.0330.2270.159
w/o PAFoMo0.1210.0300.1950.139

FoMo outperforms the simple-encoding alternative on all four metrics under both protocols. The advantage supports the benefit of the dual-geometry design over input concatenation alone under matched model capacity and training budget. We have updated Section 4.5.2 accordingly, with changes highlighted in blue. Please see the anonymous video below for qualitative comparisons.

Qualitative Comparison

W2: Runtime and peak GPU memory.

We report runtime per Euler step and peak allocated GPU memory on a single NVIDIA A800.

Fixed 5,000 vertices; varying target-frame count.

Target framesTime per Euler step (ms)Peak allocated memory (GiB)
4521.5532.998
161,649.9814.362
324,074.8516.350
487,563.8278.555
6412,186.97410.760

Fixed 16 target frames; varying vertex count.

VerticesTime per Euler step (ms)Peak allocated memory (GiB)
2,500664.4673.224
5,0001,652.9874.362
7,5002,935.4915.565
10,0004,572.8706.767

We have added detailed inference-cost measurements in Appendix E and highlighted the additions in blue. Our implementation uses the full 50-step Euler sampler, without sampling acceleration or distillation. Reducing the number of sampling steps and distilling the model offer substantial scope for faster inference. Our focus in this work is direct trajectory modeling and tracking quality; we leave these optimizations to future work.

Q2: Reference information provided in Table 2.

FoMo's direct joint prediction uses the reference mesh, its attributes, and the corresponding reference rendering, in addition to the reference skeleton. For all evaluated methods other than MoCapAnything V2, we provide the reference mesh and its corresponding rendering through each method's supported input interface. MoCapAnything V2 accepts only a skeleton as its reference-geometry input and does not support a reference mesh. It therefore receives the reference skeleton and driving video, without mesh input.

Thank you for the clarification. I will take your response, along with the comments from other reviewers, into account for my final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.