- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents EgoAhead, which builds on LANCE to jointly train current- and next-motion narration with future video and pose prediction from egocentric observations. The prediction tasks supervise shared representations during training, while narration at inference requires no future video or pose generation. On Nymeria, the authors report improved narration performance over baseline models despite using a smaller backbone.
Critical strengths
- The training–inference separation is practical: future video and pose prediction provide additional supervision during training without introducing future-generation steps when producing narrations at inference.
- Adding future-video supervision consistently improves all five next-motion narration metrics in the controlled ablation, providing convincing evidence for this part of the method.
Critical weaknesses
- The contributions of the proposed components are not sufficiently isolated. The sensor encoding and sensor-extended MaPE lack dedicated ablations, while missing +video+GRPO and text-only GRPO comparisons leave the added value of pose prediction and multimodal rewards unclear.
- Comparisons with prior methods combine published baseline scores with results on the authors’ own splits without establishing a common evaluation protocol. Clearer alignment of data splits, observation windows, target construction, and input conditions is needed to support the reported performance advantages.
Questions for authors
- You could add a +video+GRPO baseline and compare text-only versus multimodal GRPO from the same joint checkpoint to isolate the contributions of pose prediction and multimodal reward.
- How do you assess whether auxiliary prediction losses improve narration-relevant representations or cause task interference, particularly given the mixed effects of pose supervision in Table 3?
- An analysis controlling for narration length would help clarify whether the higher SBERT and lower BERTScore F1 observed after adding pose supervision reflect improved anticipation or changes in verbosity.
- Please clarify whether the published EgoLM/Ego3DLM scores and EgoAhead results use matched data splits, observation cutoffs, target definitions, and metric implementations; if they do not, how should the comparison in Table 2 be interpreted?
- Could the intermediate representations in Figure 1 be more explicitly annotated, with symbols linked to their definitions in the method section?
2 comments
Q3. An analysis controlling for narration length.
Thank you for this suggestion. In the revised paper the two metrics no longer diverge: adding pose supervision raises both SBERT and BERTScore F1 (Table 4), and the +video+pose narrations are not longer than those of +video (33.9 against 34.2 words on average). We also ran the length-controlled analysis: restricted to the Table 4 windows where the two models' narrations differ by at most three words, the gain from pose supervision remains positive on both metrics, with 95% confidence intervals that exclude zero (+0.51 [0.21, 0.82] BERTScore F1 and +0.30 [0.12, 0.49] SBERT for +video+pose over +video). The gain from video supervision also remains (+0.53 [0.06, 1.01] BERTScore F1). The improvements therefore do not come from changes in verbosity.
Q5. Could the intermediate representations in Figure 1 be more explicitly annotated?
Thank you for this suggestion. The revised Figure 1 annotates the inputs, the observed context , the noisy and predicted future latents (, , , ) and the outputs (, , , ) with the symbols defined in Sec. 3.1 and 3.3, and a legend distinguishes tokens, clean latents and noisy latents.
We thank the reviewer again for their effort and humbly request the reviewer to consider raising their score if our reply and the revised paper adequately address their concerns.
We sincerely thank Reviewer UnfC for the careful review and the constructive questions. We are pleased that the reviewer found the training-inference separation practical and the evidence for future-video supervision convincing. We have uploaded a revised paper pdf. The changes related to your concerns are marked in blue (new ablations and attribution; W1, Q1, Q2) and teal (task definition and evaluation setup, including the common protocol and narration targets; W2, Q4). Figure 1 is revised for Q5. Below we address each point.
Thank you for these suggestions. As mentioned in our response to Reviewer 3mQp W1, our previous pose prediction pathway had an implementation issue, which we have fixed, and we have further improved our pose modeling with a continuous pose representation (Sec. 4.2, Fig. 2). In our revised paper, the ablation (Table 4) adds +video+GRPO and Narration-only+GRPO, and every variant is post-trained with the same GRPO, so the reward is not a confound between variants. The added value of pose prediction is the gap between Full EgoAhead and +video+GRPO, and Full EgoAhead is better on every metric; the same holds before GRPO between +video+pose and +video (paired differences in Appendix C.5). The ablation experiment of text-only versus multimodal GRPO is still running, and we will add it to the paper in the later revision.
For sensor-extended MaPE, we are still running additional ablation experiments. We have found removing it from EgoAhead trained with future video and pose supervision lowers BLEU-1, BLEU-4 and SBERT on both narration tasks, with ROUGE-L also lower on next-motion narration and mostly unchanged on current-motion narration:
We will add the complete ablation to the paper in later revision. For the pose representation, Sec. 4.2 (Fig. 2) compares our continuous pose VAE with the VQ-VAE tokenizer of Ego4o on the same held-out windows, where ours achieves the lower reconstruction error.
Yes, they are in our revised paper. Ego3DLM released its train/test split and checkpoint in late October, which gives us the opportunity to do a fairer comparison. EgoAhead, Ego4o and Qwen3.5-4B are trained and evaluated on this released split with the same observation cutoff, the same current- and next-motion references and one metric implementation, and Ego3DLM's released checkpoint is evaluated on the same test set with the same implementation (Sec. 4.1, Table 2). EgoLM has no public code or checkpoint, so we list its scores reported on the same split by the Ego3DLM paper. Under this protocol EgoAhead is best on both narration tasks and on future-motion prediction (Tables 2 and 3). Each method's motion input is stated in Appendix C.2, and EgoAhead still outperforms Ego3DLM when the motion inputs of both are removed (Appendix C.10).
We measure both directly on the two narration tasks: predictive supervision improves next-motion narration at no cost to current-motion narration.
Benefit. In the revised paper, the ablation of each predictive objective improves every next-motion metric (Table 4), and the BERTScore F1 gains are significant under a paired bootstrap (Appendix C.5). The mixed effect of pose supervision in the submitted Table 3 came from an implementation issue in our previous pose prediction pathway, which we have fixed together with improved pose modeling (Sec. 4.2); the mixed effect is no longer present.
Interference. Current-motion narration, which does not require anticipation, stays within 0.5 points across the three SFT variants on every metric (Appendix C.5).
Representation. Per-layer CKA (Sec. 4.5, Fig. 3; Appendix C.6) shows that predictive supervision changes the backbone representation more than GRPO, consistent with the next-motion gains arising from the shared representation.