acceptodds
Under review as a conference paper at ICLR 2027

MoVi: Motion-Centric 4D Dog Reconstruction from Monocular Videos

Abstract

We present **MoVi**, a motion-centric framework for reconstructing 4D dog motion from unconstrained monocular RGB videos for quadruped imitation learning. Existing animal motion acquisition pipelines typically depend on controlled capture setups, specialized sensing, or calibrated recording environments, which limits the scale and diversity of motion data available for downstream learning. MoVi instead targets readily available in-the-wild videos and converts them into retargeting-friendly motion representations. Our framework first estimates per-frame 3D skeletons with a flow-matching-based pose model, then refines them using reprojection constraints and temporal modeling to improve consistency under monocular ambiguity and occlusion. We further introduce a trajectory predictor that recovers global root motion from articulated motion and temporal cues, producing spatially coherent 4D motion rather than appearance-oriented reconstruction. By focusing on articulated pose and global trajectory, MoVi provides smoother and more task-relevant motion representations for quadruped retargeting and imitation learning, offering a scalable way to leverage large collections of ordinary monocular videos as training data.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Temporal Motion Modeling, 4D Animal Reconstruction, Trajectory Prediction, Flow Matching

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_E1W1OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper aims to recover articulated dog motion and global root trajectories from monocular videos. The method combines an existing flow-matching-based 3D pose estimator with triggered temporal/joint refinement, scale-aware canonicalization, and a trajectory predictor consisting of a motion-prior branch, a geometry-camera residual branch, and reliability-gated fusion. Experiments evaluate pose reconstruction on Animal4D and trajectory recovery and Go2 retargeting on mocap-derived clips rendered with virtual cameras.

Critical strengths
  1. This paper addresses a practically relevant problem: extracting reference motion from monocular animal videos for quadruped imitation.
  2. It is interesting that it separates articulated pose from global trajectory.
  3. The paper is also generally easy to follow at a high level.
Critical weaknesses
  1. The experimental evidence does not sufficiently support the main in-the-wild trajectory-recovery claim.
  2. The paper does not show the contribution of each component. It is more likely to be a technical work, not a research paper.
  3. The experiments are not sufficient, neither on baselines nor on evaluation datasets.
  4. The benefit over the FMPose3D front end is limited on clean data.
  5. It seems the performance may strongly depend on the synthetic camera distribution. Not support real camera cases.
  6. Implementation details are missing.
  7. The claim is wrong. It does not perform any reconstruction.
Questions for authors

Please refer to the weaknesses.

Reviewer_2UgPOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper presents a framework for predicting 4D dog joint motion and outputs skeleton joints and root trajectory from monocular videos. Although it is framed as 4D dog reconstruction, it outputs only joint articulation and global translation, making it a paper on articulated joint learning. The pipeline uses an existing 2D-to-3D animal pose method (FMPose3D) for lifting. In practice, the results are essentially the same as FMPose3D on the Animal4D clean set, and the improvement appears only on the authors' occluded synthetic data. The authors report no results on real in-the-wild video, as they evaluate trajectory only on mocap clips with virtual cameras.

Critical strengths

The overall approach of separating joint articulation from global trajectory is reasonable. However, the paper provides insufficient information on crucial parts of the pipeline.

Critical weaknesses

The submission reads like an unfinished draft. Key information is missing throughout. The training losses used for training are only named, without definitions or weights. Data sizes, train/test splits, and how the occlusion benchmark was built are not given. Without this, the method cannot be reproduced or evaluated, and reviewers cannot judge what the results mean.

The paper motivates in-the-wild video, but all trajectory evaluations use synthetic virtual cameras, and the results are essentially the same as per-frame FMPose3D in the non-occluded case. It does not state whether the training data includes similar occluded cases or noise similar to that in the occluded cases. All baselines are general multi-species, single-frame methods, no dog-specific methods such as DigiDogs[1], BITE[2], BARC[3] or temporal baseline included. To support the claims, the paper should show that the method recovers dog motion more accurately than existing methods on real videos, measured in both camera and global coordinates.

[1] Shooter, Moira, Charles Malleson, and Adrian Hilton. "Digidogs: Single-view 3d pose estimation of dogs using synthetic training data." 2024 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW). IEEE, 2024.

[2] Rüegg, Nadine, et al. "Bite: Beyond priors for improved three-d dog pose estimation." 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2023.

[3] Rueegg, Nadine, et al. "Barc: Learning to regress 3d dog shape from images by exploiting breed information." 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2022.

Questions for authors
  1. How were the losses defined? Could you provide training details for the pipeline, including the data, splits, trajectory construction, training length, architecture type, etc.
  2. How was the occlusion benchmark constructed? Did the training data contain similar occlusions or noise?
  3. Which camera is used at test time for the trajectory? What do the GT numbers in Table 3 represent, and why is the GT error not zero? It would be helpful to report results in both 3D and 2D, and in both camera and global coordinates.
  4. Why use dog joint motion for a robot with a different body structure? If the goal is a robot motion, why limit the method to dogs rather than any quadruped?
  5. How does the downstream walking robot perform compared to other state-of-the-art robot walking methods? Can the robot learn dog-specific motion?
Reviewer_A45YOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MoVi, a framework for reconstructing articulated dog motion and global root trajectories from monocular RGB videos. The method first estimates per-frame 3D poses using FMPose3D and refines unreliable joints using temporal and inter-joint context. The refined poses are then normalized by torso length through scale-aware canonicalization.

For global trajectory recovery, the method combines a 'Motion-Prior Anchor Branch' and a 'Geometry-Camera Residual Branch' through reliability-gated fusion. The framework uses a two-stage training strategy with mocap data and virtual-camera augmentation.

The method is evaluated on the Animal4D dog subset for pose reconstruction and robustness to synthetic occlusion. Additional experiments assess trajectory recovery and Go2 retargeting using mocap-derived sequences with virtual cameras.

Critical strengths
  • The paper addresses a relevant problem of recovering articulated animal motion and global trajectories from monocular videos. The focus on motion representations for quadruped retargeting is practically motivated.

  • The method improves temporal consistency under synthetic occlusion. In Table 1, MPJVE decreases from 50.62 to 36.92 compared with FMPose3D, while mean error decreases from 58.12 to 51.15.

Critical weaknesses

[1] Insufficient methodological details and justification of the main contributions: The paper presents the 'Triggered Temporal & Joint Attention Refiner' and 'Scale-Aware Trajectory Prediction' as its main contributions, but does not sufficiently explain their novelty or justify the design choices.

For the refinement module, Figure 1 introduces 'Temporal Jump', 'Confidence Check', 'Motion State Check', and 'Anomaly Detection'. However, Section 3.3 does not explain how these signals are computed or combined to obtain the attention gate in Eq. (3). The construction of the temporal and inter-joint context is also unclear. The paper provides little discussion of how the proposed refinement differs from existing temporal refinement or confidence-based smoothing methods.

A similar issue appears in the trajectory prediction framework. Figure 2 illustrates the training pipeline, but Section 3.6 only briefly describes the training objectives. The justification for the two-stage training strategy is limited. The paper also provides little discussion of how the proposed architecture differs from existing motion-prior-based trajectory estimation methods.

These omissions make it difficult to fully understand the proposed method without referring to the implementation (source code). The novelty and rationale behind the main contributions are also not sufficiently established.

[2] Limited qualitative evaluation on in-the-wild videos: Although the paper aims to reconstruct 4D dog motion from unconstrained monocular RGB videos, Figure 3 only shows frames from a single video under clean and synthetically occluded conditions. Additional qualitative examples from different videos and motion conditions would help demonstrate the robustness of the proposed method.

[3] Insufficient comparison for the proposed temporal refinement module: In Table 1, MoVi achieves almost the same performance as FMPose3D under the clean setting. Under synthetic occlusion, MPJVE and mean error improve, but PCK remains nearly unchanged. These results show some benefit from the refinement, but do not establish whether the proposed method is better than simpler refinement approaches.

The paper does not compare the refiner with conventional temporal smoothing or confidence-weighted interpolation. Without these baselines, it is difficult to determine whether the improvements come from the proposed refinement mechanism or can be achieved through simpler temporal aggregation.

[4] Limited comparisons for the trajectory prediction framework: Table 2 shows that the individual components contribute to trajectory prediction performance. However, the paper does not compare the proposed framework with simpler alternatives that use the same input information. For example, a single-branch temporal predictor using both motion and geometry features would provide a useful reference for evaluating the proposed architecture.

Questions for authors

Please address the concerns raised in the weaknesses section, particularly regarding the methodological details and novelty of the proposed modules, comparisons with simpler baselines, and validation of global trajectory recovery.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.