acceptodds
Under review as a conference paper at ICLR 2027

Foresight to Motion: Goal-Conditioned Autoregressive Human Motion Prediction from Egocentric Video

Abstract

Generating plausible human motion from egocentric observations is important for human behavior understanding and embodied robotics. However, in first-person settings, most of the body remains outside the camera's field of view, making motion prediction from partial visual observations and egocentric camera motion challenging. Existing approaches rely on RGB-conditioned diffusion models, which are limited by sparse behavioral cues and fixed-horizon generation. To address these limitations, we introduce Foresight to Motion (FOM), a multimodal autoregressive framework for goal-driven, human motion prediction from egocentric video. Beyond visual observations, our model incorporates geometric and camera-motion cues together with explicit future interaction goals, providing guidance toward desired targets over extended temporal horizons. We further propose a three-stage training paradigm based on teacher forcing, self-forcing, and head-camera consistency, improving closed-loop robustness and reducing the train–inference discrepancy. Extensive experiments demonstrate strong performance on egocentric motion prediction benchmarks and improved cross-dataset generalization. Our goal-driven formulation also provides a flexible foundation for downstream human behavior modeling and video-conditioned motion planning for embodied systems.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Human Motion Prediction, Autoregressive Model, Multimodal modeling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_g4zUOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

FoM is a goal-conditioned autoregressive framework for predicting full-body SMPL-X motion from egocentric video. The model integrates RGB observations, estimated depth, camera trajectories, and spatial interaction goals through a decoder-only transformer. To improve autoregressive rollout stability, the authors adopt a three-stage training strategy consisting of teacher forcing, self-forcing, and head-camera consistency optimization.

Experiments on Nymeria and Aria Digital Twin (ADT) demonstrate improvements over UniEgoMotion. The authors also provide modality and training-stage ablations to analyze their contributions.

Critical strengths
  1. Promising empirical performance on egocentric motion reconstruction. FoM achieves substantial improvements over UniEgoMotion in absolute MPJPE on both Nymeria (0.304 to 0.213 m) and ADT (0.344 to 0.254 m).
  2. Easy to read and reproduce.
Critical weaknesses
  1. Ambiguous task formulation and insufficient task-appropriate evaluation. Although the paper emphasizes long-horizon motion prediction, its main evaluation appears closer to causal egocentric motion estimation, where temporally aligned RGB observations and camera trajectories are available throughout the predicted sequence. This setting differs fundamentally from forecasting future motion given only past observations.

    • If the intended contribution is future motion forecasting, the paper does not provide corresponding quantitative evaluations without future visual observations and camera trajectories.
    • Furthermore, all reported experiments use 10-second sequences, without evaluating rollouts beyond the training horizon. Therefore, the claimed advantage in long-horizon prediction remains insufficiently demonstrated.
    • Goal conditioning is a central claimed contribution, yet its effectiveness is evaluated primarily through reconstruction errors rather than goal-reaching accuracy or controllability. And the paper claims FoM is for "video-conditioned motion planning for embodied systems".
  2. Limited novelty and insufficient evidence to isolate the proposed contributions.

    • The core architecture largely follows established autoregressive motion modeling. In particular, EgoReAct [1] has already explored causal autoregressive human motion generation conditioned on egocentric visual observations, depth, and head dynamics.
    • The experimental improvements are difficult to attribute to Self Forcing.
      • In Table 3, self-forcing substantially reduces MPJPE from 0.606 to 0.194 m, while PA-MPJPE improves more modestly from 0.137 to 0.118 m. This suggests that much of the improvement is associated with global motion alignment rather than local pose accuracy. Given the availability of camera trajectories, it remains unclear whether autoregressive motion feedback provides an advantage over direct causal motion estimation.
      • Appendix Table S2 shows that self-forcing actually worsens MPJPE from 1.033 to 1.406 m when trained and evaluated on ADT, despite improving PA-MPJPE from 0.115 to 0.059 m. This raises concerns about whether self-forcing consistently improves global motion accuracy and rollout stability.
      • The main motivation for self-forcing is to mitigate the training–inference mismatch in visual conditioned AR motion generation system, but unlike [2], FoM does not update its visual observations based on the predicted motion. Therefore, self-forcing primarily addresses the exposure bias introduced by its own autoregressive design, raising questions about the necessity of autoregressive modeling in motion prediction based on multimodal obersvation.

Reference: [1] Zhang, Libo, et al. "Egoreact: Egocentric video-driven 3d human reaction generation." arXiv preprint arXiv:2512.22808 (2025). [2] Diomataris, Markos, et al. "Moving by looking: Towards vision-driven avatar motion generation." arXiv preprint arXiv:2509.19259 (2025).

Questions for authors
  1. Task formulation: FoM is motion prediction or motion reconstruction.
  2. The author should include long-horizon inference evaluation.
1 comment

We thank the reviewer for the detailed feedback and the two references.

Weakness 1 / Question 1 (task formulation). It is reconstruction, done causally. At frame t, the model sees RGB, depth, goal and camera up to frame t, plus its own earlier outputs, and estimates the pose at frame t. It never sees future frames or ground-truth poses. We used "prediction" loosely and will fix the wording.

Weakness 1 (forecasting). We do not claim forecasting. The closest setting we support is RGB from the first frame only, which we will add to Table 2. Table 2 row 2 (0.935 m without camera) already shows that the model depends on camera motion, in line with the reviewer's reading.

Weakness 1 / Question 2 (long horizon). We do not evaluate beyond the training window. By long horizon we mean a 10 s closed-loop rollout without drift, where global error accumulates (Table 3: 0.606 m without self-forcing, 0.194 m with it). We will reword the claim about extending beyond a fixed window.

Weakness 1 (goal evaluation and "motion planning").

  • The goal is a spatial anchor against drift. It tells the model where the motion ends, which keeps the global trajectory on track over 10 s. It is not meant as a separate controllability feature.
  • Reconstruction error already measures goal reaching. The goal is the ground-truth contact point, so the ground-truth hand is within 3 cm of it at contact. A predicted hand close to the ground-truth hand is also close to the goal.
  • Table 2, same weights: with the goal, MPJPE drops from 0.577 m to 0.194 m and hand error from 0.725 m to 0.347 m, while PA-MPJPE changes less (0.159 m to 0.118 m). The goal mainly corrects global placement.

We do not evaluate planning. The phrase describes a future direction, and we will reword it.

Weakness 2 (novelty vs. EgoReAct [1]). We will cite and discuss EgoReAct. It addresses a different task with a different motion representation, so the two are not directly comparable:

  • Task: EgoReAct generates a plausible reaction, where many outputs are valid. FoM reconstructs the wearer's actual motion, with one correct answer (motion capture).
  • Evaluation: realism and spatial consistency vs. metric joint error (MPJPE) over a 10 s rollout.
  • Representation: discrete VQ-VAE tokens sampled by a GPT vs. a continuous, head-centred SMPL-X representation regressed directly.
  • Problem: with a continuous representation the main difficulty is global drift, which our goal conditioning, self-forcing and head-camera consistency address.

Weakness 2 (self-forcing gain is mostly global). This is expected. Global motion is predicted as small steps and summed, so its errors grow over time, and self-forcing reduces them. Local pose is re-estimated at every frame, so PA-MPJPE changes less. The camera trajectory is a soft input, dropped in half of the training clips, so it does not remove the drift. The gain is not from extra training: teacher forcing for 400 and 500 epochs gives 0.661 m and 0.738 m (Table S1).

Weakness 2 (Table S2, ADT). This is correct. In self-forcing the loss is still per frame, so slow global drift is not penalized. On Nymeria (about 15k clips) this is not a problem. ADT is much smaller (2.4k clips, two subjects), so local pose improves but global drift grows. Stage III adds a loss on the absolute head position and reduces MPJPE on ADT from 1.406 m to 0.562 m. This is why the three-stage model is our main model.

Weakness 2 (no visual update as in [2]). We agree. In [2] the avatar's motion changes what it sees next. Our video is recorded from a real person and does not depend on our predictions. Updating it would require a simulator, which is outside our setting. So the only training–inference mismatch is in the motion history, and self-forcing addresses only that. Autoregression gives fast per-frame inference with no sampling loop and no fixed clip length.

Reviewer_yANWOpenReview ↗
Rating
2(−1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FoM autoregressively predicts full-body SMPL-X motion using egocentric RGB, depth, camera trajectories, and goals. Training combines teacher forcing, self-forcing, and head-camera consistency.

Critical strengths

Clear movitation. Modality ablations use the same weights, and Nymeria splits separate participants and locations.

Critical weaknesses
  1. The evaluation does not clearly separate online reconstruction from future forecasting, since predictions can access observations at the same timestamp. Specify the observed prefix, future-modality masking, and whether ViPE preprocessing is causal.

  2. Training goals use ground-truth contacts, but validation goals and baseline conditions are unclear. Removing goals raises MPJPE from 0.194 to 0.577 (table 2), so condition-matched baselines and tests with noisy user clicks are needed to attribute improvements. The author does not explain why the third phase is used as the main model.

Questions for authors

See weaknesses.

1 comment

We thank the reviewer for the helpful feedback. We respond to each point below.

Weakness 1. Online reconstruction vs. forecasting; observed prefix, future-modality masking, and ViPE.

  • Task. Our task is online reconstruction. At frame t, the model sees RGB, depth, goal and camera up to frame t, plus its own earlier predictions. It sees nothing after frame t. We do not evaluate or claim forecasting, and we will fix the wording in the paper.
  • Observed prefix. There is no observed motion prefix. The model starts from a learned start token and predicts all 150 frames by itself. It never receives a ground-truth pose.
  • Future-modality masking. Masking is used only in training. For half of the clips, the RGB of all frames except the first is hidden. Depth, goal and camera are each dropped for a whole clip with probability 0.5, and RGB with probability 0.1. In Tables 1–3, an input is either fully given or fully removed. We will also add the first-frame-RGB-only setting to Table 2.
  • ViPE. ViPE is not causal. It is run offline on the full clip, so its camera poses and depth at frame t also use later frames. Our model is causal, but these inputs are not. A real online system would use on-device tracking and a streaming depth model.

Weakness 2. Validation goals, baseline conditions, noisy clicks, and the choice of the third stage.

  • Validation goals. They come from the ground-truth contact point, the same as in training.
  • Baseline conditions. UniEgoMotion was retrained on our split with its original inputs and does not get depth or the goal. The extra inputs are our design choice and main contribution, so UniEgoMotion is compared as published. Table 2 already isolates each input with the same weights: with UniEgoMotion's inputs only (RGB + camera), FoM reaches 0.660 m, and adding depth and goal brings it to 0.194 m. So the gain comes from the conditioning, mainly the goal.
  • Noisy clicks. We are running a noisy-click experiment that follows Eq. 14. The goal pixel in the first frame is perturbed with Gaussian noise and lifted to 3D with the estimated depth. We will report results for several noise levels.
  • Why the third stage. Stage II has a lower MPJPE on Nymeria (0.194 m vs 0.213 m), but Stage III is more stable. In Stages I and II the loss is per frame. Global motion is stored as a small step between frames, so small errors add up over 150 frames and the loss does not see them. Stage III is the only stage with a loss on the absolute head position. This matters most with little data: on ADT, Stage II drifts to 1.406 m and Stage III recovers to 0.562 m (Table S2).
Reviewer_MntpOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Foresight to Motion (FoM), a causal autoregressive transformer that predicts full-body SMPL-X motion from egocentric video. It is conditioned on RGB (DINOv2 features), depth and camera trajectory estimated by ViPE, and a goal heatmap. The goal is a 3D interaction point projected into each frame. Modality dropout lets one model run with any subset of inputs. Training has three stages: teacher forcing, self-forcing, and an optional head-camera consistency loss. The model is trained on Nymeria and evaluated on Nymeria and on ADT (cross-dataset). The paper reports large gains over UniEgoMotion, mainly in global trajectory.

Critical strengths
  • Egocentric full-body motion prediction is an important problem for AR, avatars, and embodied robotics.
  • Modality dropout is a practical design: one checkpoint supports many input combinations, and Table 2 makes good use of this.
  • The ablation of the training strategy (Table 3) clearly shows that self-forcing matters for closed-loop rollout.
  • The cross-dataset evaluation on ADT is a welcome addition.
Critical weaknesses
  1. The autoregressive design is not well motivated for the evaluated setting. All experiments use fully observed 10-second clips, yet the model is deliberately prevented from seeing future frames. Future context is highly informative: where the camera goes next strongly constrains where the body is now. A bidirectional model with the same inputs would plausibly do better offline. Autoregression also creates its own problem, exposure bias, which the paper then needs self-forcing to fix (0.606 → 0.194 m MPJPE). A non-autoregressive model would not have this issue.

  2. No same-input baseline. The only baseline, UniEgoMotion, does not receive the depth and goal inputs that FoM uses. So the comparison cannot tell whether the gains come from the autoregressive design or simply from the extra inputs. A bidirectional (e.g., diffusion or transformer) baseline with identical inputs (RGB, depth, camera, goal) is needed to justify the main design choice.

  3. The motivation and the evaluation do not match. The paper motivates autoregression and the “foresight” framing with streaming/online use, long horizons beyond a fixed window, and future frames filled by a video model (Wan 2.2). None of these is evaluated. There is no latency or streaming evaluation, no sequence longer than 150 frames, and no experiment with generated video. As it stands, the title’s central idea is a stated possibility, not a demonstrated result.

  4. The source of goals at evaluation time is unspecified, and may be ground truth. Section 3.3 describes an inference interface where a user clicks a pixel in the first frame. However, the paper never says how goals were obtained for the reported experiments. It mentions no annotators clicking the 3,439 validation clips, no detector, and no gaze-based method. The ADT clips are described as “contact-centered,” which suggests the goals come from ground-truth contact annotations, as in training. If so, the model is told where the hand will make contact, which is privileged information that the baseline does not have. This is a serious concern, and on its own it calls the main comparison into question. For goal conditioning to be practical, goals should come from a realistic source, such as a high-level planner or vision-language model, gaze, or noisy user clicks, and performance should be reported under those conditions.

  5. Without the goal, FoM is substantially worse than the baseline. Table 2 row 3 (RGB + depth + camera, no goal) gives MPJPE 0.577 m. UniEgoMotion’s MPJPE in Table 1 is 0.304 m. FoM without the goal loses on every metric. This is despite FoM still using depth and camera trajectory. It seems that the most gains depend entirely on the goal input.

  6. Goal visibility is not addressed. The inference interface requires the goal to be visible in the first frame with valid depth. During training, however, goals are defined in world coordinates from future contact, so they can start off-screen or occluded. It is unclear what happens at inference when the target is not visible in frame 1. It is also unclear how the heatmap is defined when the projected goal leaves the field of view, and how often either case occurs in the training and evaluation data.

Overall, the modality-flexible design and the self-forcing analysis are useful. However, the main claims are not supported by the experiments: the benefit of autoregression is never isolated, the motivating settings are never evaluated, and the improvement over the baseline appears to depend on a goal input whose source is unspecified and may be ground truth. I am happy to change my ratings if the question 1-4 are addressed in the rebuttal.

Questions for authors
  1. How exactly were goals obtained for every result in Tables 1–3 and the figures? Were they derived from ground-truth contact?
  2. How does FoM perform with realistic goals (noisy clicks, gaze, or a detector/VLM), and with no goal, compared with UniEgoMotion?
  3. Can you provide a bidirectional baseline with identical inputs?
  4. What fraction of training and evaluation goals are visible in the first frame? How are out-of-view goals represented in the heatmap?
1 comment

We thank the reviewer for the detailed feedback and respond to each point below.

Weakness 1 (autoregressive design). We agree that a bidirectional model should do better offline. Our target is real-world online use (robotics, wearable devices), where the pose at time t must be produced at time t, so the model is causal. We use recorded clips only because they provide ground truth. Autoregression costs one extra fine-tuning stage (self-forcing) and gives a single forward pass per frame, no sampling loop, and no fixed clip length. We will state this setting clearly.

Weakness 2 (no same-input baseline). The extra inputs are our design choice and main contribution, so we compare with UniEgoMotion as published.

  • Depth is not an extra sensor. It is estimated from the same RGB video.
  • UniEgoMotion has no interface for a spatial goal. Adding one means designing a new model.
  • Table 2 isolates each input with the same weights: 0.660 m with UniEgoMotion's inputs (RGB + camera), 0.577 m with depth added, 0.194 m with the goal added.

So the gain comes from the conditioning, mainly the goal.

Weakness 3 (motivation vs. evaluation). We agree that the framing goes beyond what we evaluate, and we will revise it.

  • Online use: evaluation is causal and closed-loop over all 150 frames, with no ground-truth pose. We will add per-frame latency.
  • Long horizon: we mean a 10 s rollout without drift (Table 3: 0.606 m without self-forcing, 0.194 m with it). We do not evaluate beyond 150 frames and will reword that claim.
  • Generated video: we have no Wan 2.2 experiment and will present it only as future work.

"Foresight" refers to the goal, which is known before the motion starts.

Weakness 4 / Question 1 (goal source). Yes. For all results in Tables 1–3 and the figures, in training and evaluation, the goal is the ground-truth contact point (Eq. 13) projected into each frame. No annotator, detector, or gaze is used. We will state this in the paper. The goal is a single 3D point: it gives the end location of the hand, not the timing, path, or pose. It is our main design choice, a spatial anchor that keeps the global trajectory on track.

Weakness 5 (no goal). We agree that the goal is the main source of the gain. This is the intended design, and the paper reports it: removing the goal raises MPJPE from 0.194 m to 0.577 m (Table 2, Fig. S3). The paper did not state that this is worse than UniEgoMotion (0.304 m), or that UniEgoMotion receives no goal or depth. Two reasons explain the gap. First, UniEgoMotion processes its full window at once (8 s, its maximum), including the future head trajectory, while FoM sees only past and current frames. Second, row 3 is the same model with the goal hidden at test time on goal-reaching clips, not a model trained without goals. The camera trajectory is a soft input, dropped in half of the training clips, so it does not replace the goal.

Question 2 (realistic goals). All reported results use ground-truth goals. We have not evaluated noisy clicks, gaze, or detector/VLM goals, and we will state this as a limitation. The interface in Eq. 14 accepts a clicked pixel in the first frame, lifted to 3D with the estimated depth, but its accuracy under noisy goals is not measured in this work. For the no-goal case, see Weakness 5.

Question 3 (bidirectional baseline). We do not provide one. It needs the full clip, so it does not fit the causal, real-world setting we target. We train and evaluate the model the way it would run in deployment. Table 2 isolates the effect of each input (Weakness 2).

Weakness 6 / Question 4 (goal visibility). 100% of training and evaluation goals are visible in the first frame, by construction: each clip starts at the first frame where the goal is inside the camera view. Goals that start off-screen do not occur in our data, so the click interface applies to every clip. If the goal leaves the view later, for example when the person looks away, the heatmap for that frame is all zeros, and the model sees such frames in training. Visibility is checked against the camera view, not occlusion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.