acceptodds
Withdrawn from ICLR 2027

EgoAhead: Egocentric Human Motion Understanding with Joint Video–Motion Prediction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Understanding and anticipating human motion from egocentric observations is a key task for proactive AR/XR and embodied assistants. Existing egocentric behavior understanding models are typically trained only to describe the current motion, while methods that forecast future motion rely on either hints from the future recording or precomputed 3D environment scene features, limiting their applicability in real-world settings. To address this gap, we introduce EgoAhead, a unified multimodal model trained jointly for motion description and future video and pose prediction. Extending from an unified backbone Lance, EgoAhead allows multimodal generation of future video frames and poses to co-supervise a shared representation alongside language-based motion description during training. At inference, the model can output language descriptions without generating video or pose tokens, preserving inference efficiency. On the Nymeria benchmark, EgoAhead outperforms baseline models on both current-motion and future-motion descriptions despite using a smaller backbone. Our findings demonstrate that multimodal generation supervision provides an effective way to strengthen egocentric behavior understanding for wearable AI assistants.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Egocentric Human Motion Understanding, Smart Glasses, Unified Multimodal Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UnfCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents EgoAhead, which builds on LANCE to jointly train current- and next-motion narration with future video and pose prediction from egocentric observations. The prediction tasks supervise shared representations during training, while narration at inference requires no future video or pose generation. On Nymeria, the authors report improved narration performance over baseline models despite using a smaller backbone.

Critical strengths
  1. The training–inference separation is practical: future video and pose prediction provide additional supervision during training without introducing future-generation steps when producing narrations at inference.
  2. Adding future-video supervision consistently improves all five next-motion narration metrics in the controlled ablation, providing convincing evidence for this part of the method.
Critical weaknesses
  1. The contributions of the proposed components are not sufficiently isolated. The sensor encoding and sensor-extended MaPE lack dedicated ablations, while missing +video+GRPO and text-only GRPO comparisons leave the added value of pose prediction and multimodal rewards unclear.
  2. Comparisons with prior methods combine published baseline scores with results on the authors’ own splits without establishing a common evaluation protocol. Clearer alignment of data splits, observation windows, target construction, and input conditions is needed to support the reported performance advantages.
Questions for authors
  1. You could add a +video+GRPO baseline and compare text-only versus multimodal GRPO from the same joint checkpoint to isolate the contributions of pose prediction and multimodal reward.
  2. How do you assess whether auxiliary prediction losses improve narration-relevant representations or cause task interference, particularly given the mixed effects of pose supervision in Table 3?
  3. An analysis controlling for narration length would help clarify whether the higher SBERT and lower BERTScore F1 observed after adding pose supervision reflect improved anticipation or changes in verbosity.
  4. Please clarify whether the published EgoLM/Ego3DLM scores and EgoAhead results use matched data splits, observation cutoffs, target definitions, and metric implementations; if they do not, how should the comparison in Table 2 be interpreted?
  5. Could the intermediate representations in Figure 1 be more explicitly annotated, with symbols linked to their definitions in the method section?
2 comments

We sincerely thank Reviewer UnfC for the careful review and the constructive questions. We are pleased that the reviewer found the training-inference separation practical and the evidence for future-video supervision convincing. We have uploaded a revised paper pdf. The changes related to your concerns are marked in blue (new ablations and attribution; W1, Q1, Q2) and teal (task definition and evaluation setup, including the common protocol and narration targets; W2, Q4). Figure 1 is revised for Q5. Below we address each point.


W1 / Q1. The contributions of the components are not sufficiently isolated: +video+GRPO, text-only versus multimodal GRPO, and ablations of the sensor encoding and sensor-extended MaPE.

Thank you for these suggestions. As mentioned in our response to Reviewer 3mQp W1, our previous pose prediction pathway had an implementation issue, which we have fixed, and we have further improved our pose modeling with a continuous pose representation (Sec. 4.2, Fig. 2). In our revised paper, the ablation (Table 4) adds +video+GRPO and Narration-only+GRPO, and every variant is post-trained with the same GRPO, so the reward is not a confound between variants. The added value of pose prediction is the gap between Full EgoAhead and +video+GRPO, and Full EgoAhead is better on every metric; the same holds before GRPO between +video+pose and +video (paired differences in Appendix C.5). The ablation experiment of text-only versus multimodal GRPO is still running, and we will add it to the paper in the later revision.

For sensor-extended MaPE, we are still running additional ablation experiments. We have found removing it from EgoAhead trained with future video and pose supervision lowers BLEU-1, BLEU-4 and SBERT on both narration tasks, with ROUGE-L also lower on next-motion narration and mostly unchanged on current-motion narration:

Gain from sensor-extended MaPEBLEU-1BLEU-4ROUGE-LSBERT
Current-motion narration+6.69+1.99−0.02+0.73
Next-motion narration+7.58+2.69+0.50+1.61

We will add the complete ablation to the paper in later revision. For the pose representation, Sec. 4.2 (Fig. 2) compares our continuous pose VAE with the VQ-VAE tokenizer of Ego4o on the same held-out windows, where ours achieves the lower reconstruction error.

W2 / Q4. Do the EgoLM/Ego3DLM scores and EgoAhead results use matched data splits, observation cutoffs, target definitions and metric implementations?

Yes, they are in our revised paper. Ego3DLM released its train/test split and checkpoint in late October, which gives us the opportunity to do a fairer comparison. EgoAhead, Ego4o and Qwen3.5-4B are trained and evaluated on this released split with the same observation cutoff, the same current- and next-motion references and one metric implementation, and Ego3DLM's released checkpoint is evaluated on the same test set with the same implementation (Sec. 4.1, Table 2). EgoLM has no public code or checkpoint, so we list its scores reported on the same split by the Ego3DLM paper. Under this protocol EgoAhead is best on both narration tasks and on future-motion prediction (Tables 2 and 3). Each method's motion input is stated in Appendix C.2, and EgoAhead still outperforms Ego3DLM when the motion inputs of both are removed (Appendix C.10).

Q2. How do you assess whether auxiliary prediction losses improve narration-relevant representations or cause task interference?

We measure both directly on the two narration tasks: predictive supervision improves next-motion narration at no cost to current-motion narration.

Benefit. In the revised paper, the ablation of each predictive objective improves every next-motion metric (Table 4), and the BERTScore F1 gains are significant under a paired bootstrap (Appendix C.5). The mixed effect of pose supervision in the submitted Table 3 came from an implementation issue in our previous pose prediction pathway, which we have fixed together with improved pose modeling (Sec. 4.2); the mixed effect is no longer present.

Interference. Current-motion narration, which does not require anticipation, stays within 0.5 points across the three SFT variants on every metric (Appendix C.5).

Representation. Per-layer CKA (Sec. 4.5, Fig. 3; Appendix C.6) shows that predictive supervision changes the backbone representation more than GRPO, consistent with the next-motion gains arising from the shared representation.

Q3. An analysis controlling for narration length.

Thank you for this suggestion. In the revised paper the two metrics no longer diverge: adding pose supervision raises both SBERT and BERTScore F1 (Table 4), and the +video+pose narrations are not longer than those of +video (33.9 against 34.2 words on average). We also ran the length-controlled analysis: restricted to the Table 4 windows where the two models' narrations differ by at most three words, the gain from pose supervision remains positive on both metrics, with 95% confidence intervals that exclude zero (+0.51 [0.21, 0.82] BERTScore F1 and +0.30 [0.12, 0.49] SBERT for +video+pose over +video). The gain from video supervision also remains (+0.53 [0.06, 1.01] BERTScore F1). The improvements therefore do not come from changes in verbosity.

Q5. Could the intermediate representations in Figure 1 be more explicitly annotated?

Thank you for this suggestion. The revised Figure 1 annotates the inputs, the observed context CC, the noisy and predicted future latents (zτpz^p_\tau, zτvz^v_\tau, z^0p\widehat{z}^p_0, z^0v\widehat{z}^v_0) and the outputs (P+P^+, X+X^+, y−y^-, y+y^+) with the symbols defined in Sec. 3.1 and 3.3, and a legend distinguishes tokens, clean latents and noisy latents.


We thank the reviewer again for their effort and humbly request the reviewer to consider raising their score if our reply and the revised paper adequately address their concerns.

Reviewer_e4VMOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies a assistant that describes the person's current motion and anticipates the next one in language. Its inputs are egocentric video, body pose and IMU. The motivation is that learning to predict future observations may provide representations useful for narration, beyond training only on language outputs. EgoAhead builds on a unified multimodal backbone. During training it narrates current and next motion while predicting the next two seconds of egocentric video and pose. An attention mask prevents narration from attending to the future targets, so inference does not require sampling those outputs. A second stage uses multimodal GRPO with text, pose and video rewards. On Nymeria, the paper reports improved narration relative to the compared models and presents ablations of predictive supervision and reinforcement learning.

Critical strengths

The next-motion narration task is well motivated. A proactive assistant needs to anticipate what its user may do, rather than only describe an observed action. The paper develops a concrete version of this problem on Nymeria and combines it with current-motion narration in one model. It uses egocentric observations without an explicit 3D scene representation. This is the paper's main task contribution.

Future-video supervision improves next-motion narration without generation at inference. Adding video prediction improves all five next-motion metrics over Narration-only in Table 3; BLEU-1 rises from 35.95 to 38.45. The attention design separates narration from future targets, and the model does not need to generate video or pose at inference. This makes auxiliary future-video prediction a practical training recipe.

Critical weaknesses

The headline results are not attributable to the proposed supervision The paper's own Narration-only variant, with no future prediction and no reinforcement stage, already beats every baseline in Table 2 on all five current-motion metrics, and on current motion the full model is no better than it (BLEU-1 drops by 6.5 points). What the experiments support is narrower: future-video supervision improves next-motion narration, while the effects of pose prediction and of the reinforcement stage are mixed and not isolated.

The broad motivation overlooks existing predictive approaches. EgoAgent (ICCV 2025) already connects future egocentric-state and body-motion prediction with a shared representation. EgoLM (CVPR 2025), a baseline here, also links motion generation and narration. Moreover, EgoAgent and UniEgoMotion (ICCV 2025) forecast from past observations without future hints or an explicit 3D scene. These are counterexamples to the abstract's broad description of forecasting methods.

Questions for authors

Where does the 22-joint pose come from at inference? Are train/test splits separated by participant, recording or window, and how is overlap between nearby windows handled?

Can the available baselines, Narration-only and +video be evaluated on one split with each method's actual input setting stated? Please clarify the Ego4o re-evaluation as well.

What do a pose-only variant and GRPO applied to Narration-only or +video achieve? These controls would separate the benefit of prediction, body-pose supervision and multimodal reinforcement.

1 comment

We sincerely thank Reviewer e4VM for the detailed and thoughtful review. We are pleased that the reviewer found the next-motion narration task well motivated and auxiliary future-video prediction a practical training recipe. We have uploaded a revised paper pdf. The changes related to your concerns are marked in blue (attribution of the gains, new ablations and positioning against prior work; W1, W2, Q3) and teal (task definition and evaluation setup, including input settings, splits and baselines; Q1, Q2). Below we address each point.


W1. The headline results are not attributable to the proposed supervision; the effects of pose prediction and of the reinforcement stage are mixed and not isolated.

Thank you for this point. As mentioned in our response to Reviewer 3mQp W1, our previous pose prediction pathway had an implementation issue, which we have fixed, and we have further improved our pose modeling with a continuous pose representation (Sec. 4.2, Fig. 2). In our revised paper, we isolate each component, and the claims follow the evidence. The comparison with prior methods is made under one protocol (Sec. 4.1, Table 2). The contribution of each component is isolated by a controlled ablation with identical inputs, backbone, narration targets and decoding settings (Table 4): future-video supervision improves all five next-motion metrics, future-pose supervision improves every metric further, and GRPO adds further gains while preserving this ordering. The effects are no longer mixed. Accordingly, the revised paper attributes the benefit of predictive supervision to next-motion narration, the task that requires anticipation; current-motion narration is comparable across the variants (Sec. 4.4, Appendix C.5).

W2. The motivation overlooks existing predictive approaches (EgoAgent, EgoLM, UniEgoMotion).

We apologize for the oversight. We revised the abstract, the introduction, Sec. 2 and Table 1. The introduction now states that EgoAgent and UniEgoMotion forecast from past observations alone, with body-pose trajectories or latent visual features as outputs, and that EgoLM narrates jointly with motion tracking. None of these methods narrates the wearer's future behavior, which is the setting EgoAhead addresses.

Q1. Where does the 22-joint pose come from at inference? How are the splits separated, and how is overlap between nearby windows handled?

In Nymeria each participant wears a motion-capture suit, Project Aria glasses and two wristbands, and EgoAhead takes the full-body pose and inertial signals recorded in the five seconds before t0 (Sec. 4.2, Appendix C.2). To understand the impact of this input, we also evaluate EgoAhead and Ego3DLM with their motion inputs removed: with the motion inputs of both models removed, EgoAhead still outperforms Ego3DLM on every metric for both current- and next-motion narration (Appendix C.10). We discuss simplifying the input to three-point tracking in the limitations (Sec. 5). Training and test data are separated at the level of whole recordings, so overlapping windows never fall on both sides of a split; on full Nymeria, all held-out windows come from locations and participants unseen during training (Sec. 4.1).

Q2. Can the baselines, Narration-only and +video be evaluated on one split with each method's input setting stated? Please clarify the Ego4o re-evaluation.

Yes. Ego4o is trained with its official code on the same training data and narration references as EgoAhead, and is evaluated on the same test data with the same metric implementation. Ego3DLM released its train/test split and checkpoint in late October, which gives us the opportunity for a fairer comparison: Table 2 evaluates all baselines and EgoAhead on this Ego3DLM released split, and Table 4 evaluates the models trained on it together with Narration-only, +video and the other variants on one common test set. EgoAhead is the best method in both tables. Each method's motion input is stated in Appendix C.2.

Q3. What do a pose-only variant and GRPO applied to Narration-only or +video achieve?

GRPO applied to Narration-only and to +video is now reported in Table 4. Both remain below Full EgoAhead on every metric, so GRPO alone does not replace predictive supervision. The effect of pose supervision is isolated by adding it to +video under identical settings, where it improves every next-motion metric both before and after GRPO. We are also training the pose-only variant and will add it to the paper in later revision.


We thank the reviewer again for their effort and humbly request the reviewer to consider raising their score if our reply and the revised paper adequately address their concerns.

Reviewer_3mQpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The proposed method merged video, pose, and IMU inputs for motion prediction. It uses future video/pose prediction and GRPO during training, and reports improved narration results on Nymeria dataset.

Critical strengths
  • The motivation makes sense to me. Predicting future video and motion could help the model understand what happens next.
  • Adding video prediction improves all five next-motion metrics. The model can also generate narration without running future video generation.
Critical weaknesses
  • Unclear benefit of pose prediction: Adding pose prediction hurts four of five next-motion metrics in Table 3. Why? I would like comparisons with +video+GRPO and narration-only+GRPO.
  • Different test splits: Some baseline scores come from different splits. Are these results comparable? It will be good to clarify whether training and test recordings and participants overlap.
  • Confusing task definition: The input ends when the “current” action starts. What is is this current-motion understanding? Does the next action fall within the two-second prediction window?
  • Limited novelty: The method mostly adds existing components to LANCE. What is the main technical contribution? I would like an ablation on sensor-extended MaPE.
  • Narration evaluation: GRPO directly optimise SBERT and BERTScore. Is it possible that there exist any reward hacking? I would like to see a detailed qualitative analysis on that.
  • Benchmark: It will be nice to see results on more benchmarks, especially for in-the-wild episodes.
Questions for authors

N/A

2 comments

We sincerely thank Reviewer 3mQp for their thoughtful comments. We are pleased that the reviewer found the motivation sound and recognized that future-video supervision improves next-motion narration without future generation at inference. We have uploaded a revised paper pdf. The changes related to your concerns are marked in blue (new ablations, new benchmark and positioning; W1, W4, W6) and teal (task definition and evaluation setup; W2, W3, W5). Below we address each point.


W1. Unclear benefit of pose prediction. I would like comparisons with +video+GRPO and narration-only+GRPO.

Thank you for raising this issue. Our previous pose prediction pathway had an implementation issue, which we have fixed, and we have further improved our pose modeling with a continuous pose representation (Sec. 4.2, Fig. 2). With current pose supervision, adding future-pose prediction improves all five next-motion metrics over +video. We also added the requested Narration-only+GRPO and +video+GRPO baselines: the ordering narration-only < +video < +video+pose holds on every metric both before and after GRPO, and Full EgoAhead achieves the best result in every column (Table 4; paired differences in Appendix C.5).

W2. Different test splits. Are these results comparable? Do training and test recordings and participants overlap?

Ego3DLM released its train/test split and checkpoint in late October, which gives us the opportunity for a fairer comparison. In the revision all methods are trained and evaluated on Ego3DLM’s released split (Sec. 4.1), and EgoAhead achieves the best results on current-motion narration, next-motion narration and future-motion prediction (Tables 2 and 3). We follow the released split as is, and its training and test recordings do not overlap. We additionally further evaluate the methods on the full Nymeria dataset, where all held-out windows come from locations and participants unseen during training (Table 4).

W3. Confusing task definition. What is current-motion understanding? Does the next action fall within the two-second prediction window?

We agree the task definition was confusing and have revised the definition accordingly (Sec. 3.1, Appendix C.1). We have revised the definition (Sec. 3.1, Appendix C.1). The observed input ends at t0t_0. Current-motion narration describes the action that ends at t0t_0, i.e., the action just observed, and next-motion narration describes the action that starts at t0t_0. The future video and pose targets cover [t0,t0+2 s][t_0, t_0+2\,\mathrm{s}], so yes, the prediction window spans the beginning of the next action. All results in our revised paper follow this definition.

W4. Limited novelty. What is the main technical contribution? I would like an ablation on sensor-extended MaPE.

LANCE is a video-and-text model with no body-motion input and no motion output. EgoAhead adds both: a motion encoder with sensor-extended MaPE places pose and IMU signals on the same physical timeline as the video (Sec. 3.2), and a flow-matching pathway predicts future motion in a continuous pose latent space, roughly halving the position error of Ego3DLM (Table 3). Its central idea is to use future video and pose generation as supervision for narration during training only, so narration needs no generation at inference. To our knowledge, EgoAhead is the first model to combine language narration with future egocentric-video and body-motion prediction from observed wearable inputs alone, and the first to show that both forms of predictive supervision improve next-motion narration (Sec. 5).

For sensor-extended MaPE, we are still running additional ablation experiments. We have found removing it from EgoAhead trained with future video and pose supervision lowers BLEU-1, BLEU-4 and SBERT on both narration tasks, with ROUGE-L also lower on next-motion narration and mostly unchanged on current-motion narration:

Gain from sensor-extended MaPEBLEU-1BLEU-4ROUGE-LSBERT
Current-motion narration+6.69+1.99−0.02+0.73
Next-motion narration+7.58+2.69+0.50+1.61

We will add the complete ablation to the paper in later revision.

W5. GRPO directly optimises SBERT and BERTScore. Is reward hacking possible? I would like a qualitative analysis.

Thank you for this point and we have revised our GRPO reward design. In our revised paper, for all GRPO experiments, the GRPO reward uses none of the reported metrics: it based on the cosine similarity of EmbeddingGemma 2 embeddings, a text encoder distinct from those used by the evaluation metrics (Sec. 3.4; full form in Appendix B.4). GRPO still improves BLEU-4, ROUGE-L, SBERT and BERTScore F1 for every variant (Table 4), none of which it optimizes, so the gains do not come from fitting the evaluation metrics. A qualitative example with the narrations of all variants is given in Appendix C.8 (Fig. 5, Table 10).

W6. Results on more benchmarks, especially in-the-wild episodes.

We added a second benchmark. On Ego-Exo4D, an unseen dataset of skilled human activities, we evaluate all models zero-shot, without any fine-tuning (Sec. 4.3, Appendix C.9). With the egocentric video as its only input, EgoAhead outperforms Qwen3.5-4B and Ego3DLM with three-point tracking, all trained on the same split, on every metric for both current- and next-motion narration (e.g., next-motion SBERT 58.31 against 45.41 for Qwen3.5-4B, the strongest baseline).

Within Nymeria, we also added a stricter test of generalization (Table 4). The models trained on the Ego3DLM split, which covers about 8% of Nymeria, are also evaluated on held-out recordings from the rest of the dataset, whose locations and participants never appear in their training data. EgoAhead outperforms Ego3DLM and Qwen3.5-4B on every metric there.


We thank the reviewer again for their effort and humbly request the reviewer to consider raising their score if our reply and the revised paper adequately address their concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.