- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
FoM is a goal-conditioned autoregressive framework for predicting full-body SMPL-X motion from egocentric video. The model integrates RGB observations, estimated depth, camera trajectories, and spatial interaction goals through a decoder-only transformer. To improve autoregressive rollout stability, the authors adopt a three-stage training strategy consisting of teacher forcing, self-forcing, and head-camera consistency optimization.
Experiments on Nymeria and Aria Digital Twin (ADT) demonstrate improvements over UniEgoMotion. The authors also provide modality and training-stage ablations to analyze their contributions.
Critical strengths
- Promising empirical performance on egocentric motion reconstruction. FoM achieves substantial improvements over UniEgoMotion in absolute MPJPE on both Nymeria (0.304 to 0.213 m) and ADT (0.344 to 0.254 m).
- Easy to read and reproduce.
Critical weaknesses
-
Ambiguous task formulation and insufficient task-appropriate evaluation. Although the paper emphasizes long-horizon motion prediction, its main evaluation appears closer to causal egocentric motion estimation, where temporally aligned RGB observations and camera trajectories are available throughout the predicted sequence. This setting differs fundamentally from forecasting future motion given only past observations.
- If the intended contribution is future motion forecasting, the paper does not provide corresponding quantitative evaluations without future visual observations and camera trajectories.
- Furthermore, all reported experiments use 10-second sequences, without evaluating rollouts beyond the training horizon. Therefore, the claimed advantage in long-horizon prediction remains insufficiently demonstrated.
- Goal conditioning is a central claimed contribution, yet its effectiveness is evaluated primarily through reconstruction errors rather than goal-reaching accuracy or controllability. And the paper claims FoM is for "video-conditioned motion planning for embodied systems".
-
Limited novelty and insufficient evidence to isolate the proposed contributions.
- The core architecture largely follows established autoregressive motion modeling. In particular, EgoReAct [1] has already explored causal autoregressive human motion generation conditioned on egocentric visual observations, depth, and head dynamics.
- The experimental improvements are difficult to attribute to Self Forcing.
- In Table 3, self-forcing substantially reduces MPJPE from 0.606 to 0.194 m, while PA-MPJPE improves more modestly from 0.137 to 0.118 m. This suggests that much of the improvement is associated with global motion alignment rather than local pose accuracy. Given the availability of camera trajectories, it remains unclear whether autoregressive motion feedback provides an advantage over direct causal motion estimation.
- Appendix Table S2 shows that self-forcing actually worsens MPJPE from 1.033 to 1.406 m when trained and evaluated on ADT, despite improving PA-MPJPE from 0.115 to 0.059 m. This raises concerns about whether self-forcing consistently improves global motion accuracy and rollout stability.
- The main motivation for self-forcing is to mitigate the training–inference mismatch in visual conditioned AR motion generation system, but unlike [2], FoM does not update its visual observations based on the predicted motion. Therefore, self-forcing primarily addresses the exposure bias introduced by its own autoregressive design, raising questions about the necessity of autoregressive modeling in motion prediction based on multimodal obersvation.
Reference: [1] Zhang, Libo, et al. "Egoreact: Egocentric video-driven 3d human reaction generation." arXiv preprint arXiv:2512.22808 (2025). [2] Diomataris, Markos, et al. "Moving by looking: Towards vision-driven avatar motion generation." arXiv preprint arXiv:2509.19259 (2025).
Questions for authors
- Task formulation: FoM is motion prediction or motion reconstruction.
- The author should include long-horizon inference evaluation.
We thank the reviewer for the detailed feedback and the two references.
Weakness 1 / Question 1 (task formulation). It is reconstruction, done causally. At frame t, the model sees RGB, depth, goal and camera up to frame t, plus its own earlier outputs, and estimates the pose at frame t. It never sees future frames or ground-truth poses. We used "prediction" loosely and will fix the wording.
Weakness 1 (forecasting). We do not claim forecasting. The closest setting we support is RGB from the first frame only, which we will add to Table 2. Table 2 row 2 (0.935 m without camera) already shows that the model depends on camera motion, in line with the reviewer's reading.
Weakness 1 / Question 2 (long horizon). We do not evaluate beyond the training window. By long horizon we mean a 10 s closed-loop rollout without drift, where global error accumulates (Table 3: 0.606 m without self-forcing, 0.194 m with it). We will reword the claim about extending beyond a fixed window.
Weakness 1 (goal evaluation and "motion planning").
We do not evaluate planning. The phrase describes a future direction, and we will reword it.
Weakness 2 (novelty vs. EgoReAct [1]). We will cite and discuss EgoReAct. It addresses a different task with a different motion representation, so the two are not directly comparable:
Weakness 2 (self-forcing gain is mostly global). This is expected. Global motion is predicted as small steps and summed, so its errors grow over time, and self-forcing reduces them. Local pose is re-estimated at every frame, so PA-MPJPE changes less. The camera trajectory is a soft input, dropped in half of the training clips, so it does not remove the drift. The gain is not from extra training: teacher forcing for 400 and 500 epochs gives 0.661 m and 0.738 m (Table S1).
Weakness 2 (Table S2, ADT). This is correct. In self-forcing the loss is still per frame, so slow global drift is not penalized. On Nymeria (about 15k clips) this is not a problem. ADT is much smaller (2.4k clips, two subjects), so local pose improves but global drift grows. Stage III adds a loss on the absolute head position and reduces MPJPE on ADT from 1.406 m to 0.562 m. This is why the three-stage model is our main model.
Weakness 2 (no visual update as in [2]). We agree. In [2] the avatar's motion changes what it sees next. Our video is recorded from a real person and does not depend on our predictions. Updating it would require a simulator, which is outside our setting. So the only training–inference mismatch is in the motion history, and self-forcing addresses only that. Autoregression gives fast per-frame inference with no sampling loop and no fixed clip length.