acceptodds
Withdrawn from ICLR 2027

EgoForge: Goal-Directed Egocentric World Simulator

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Generative world models have shown promise for simulating dynamic environments, yet egocentric video remains challenging due to rapid viewpoint changes, frequent hand-object interactions, and goal-directed procedures whose evolution depends on latent human intent. Existing approaches either focus on hand-centric instructional synthesis with limited scene evolution, perform static view translation without modeling action dynamics, or rely on dense supervision, such as camera trajectories, long video prefixes, synchronized multi-camera capture, In this work, we introduce EgoForge, an egocentric goal-directed world simulator that generates coherent, first-person video rollouts from minimal static inputs: a single egocentric image, a high-level instruction, and an optional auxiliary exocentric view. To improve intent alignment and temporal coherence, we introduce VideoDiffusionNFT, a trajectory-level diffusion refinement method that uses positive and negative rollout distributions, derived from goal, temporal, scene-consistency, and perceptual rewards, to steer the diffusion velocity field toward coherent, goal-complete egocentric simulations. Extensive experiments show EgoForge achieves consistent gains in semantic alignment, geometric stability, and motion fidelity over strong baselines, and robust performance in real-world smart-glasses experiments.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Computer vision, Video generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_TKXROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

EgoForge generates goal-directed egocentric video rollouts from minimal inputs: a single egocentric frame, a high-level instruction, and an optional exocentric reference image, with no camera trajectories, pose signals, video prefixes, or synchronized multi-view capture at inference. It is built on a Wan2.2-5B diffusion-transformer backbone with flow matching. The paper also curates X-Ego, 15,000 training clips plus 100 held-out test clips from Nymeria and Ego-Exo4D with hand-object annotations. Seven automatic metrics against five baselines plus matched-conditioning fine-tuned variants, component ablations, counterfactual conditioning tests, instruction sensitivity, a 20-rater human study, real-world ARGO smart-glasses deployment, and judge-reliability and reward-hacking analyses.

Critical strengths

The evaluation is thorough. Matched-conditioning baselines (+EV/+TT/+CI), an extra-SFT training-budget control, counterfactual conditioning tests, clip-start vs. clip-end exo references, instruction sensitivity, reward ablations, cross-judge and human agreement, a reward-hacking probe, and cross-backbone GRAFT transfer (Cosmos, HunyuanVideo) together make the results largely believable. The transparency about Wan2.2+CI being the authors' own Stage-1 and the separation of full-system versus method-only gains (Table 9) is exactly how this should be reported.

Critical weaknesses
  1. The headline numbers rely on an exocentric reference taken from the end of the clip, which reveals the final object state, i.e., the answer to the goal-completion question. The paper acknowledges this (Appendix H.6) and provides clip-start and no-exo controls showing the gains persist, which mitigates the concern. But Tables 2-4 and the abstract-level claims are built on the clip-end setting. If "goal-directed generation from minimal static inputs" is the claim, the clip-start or no-reference results should be the primary tables; otherwise the claim is weaker than presented.
  2. The evaluation scale is small: only 100 held-out test clips, 20 raters over 25 groups, and K=6 rollouts per condition for GRAFT training. FVD and flow MSE on 100 clips are noisy, and no variances or confidence intervals are reported. The "first large-scale benchmark" characterization of X-Ego also sits awkwardly on a 15k-sample derivative of Nymeria and Ego-Exo4D with LLM-refined captions, and the test set is much smaller than the training set.
Questions for authors
  1. Could the clip-start (or no-exo) results be made the primary tables? Given that the goal reward and the clip-end reference both carry future information, how much of the reported gains over baselines in Tables 2-3 depends on seeing the final state?
  2. With 100 test clips, what are the variances or confidence intervals of FVD, flow MSE, and DINO-Score? Are the reported gains statistically meaningful?
  3. At deployment (e.g., the smart-glasses scenario) there is no target-state image. How is Rgoal operationalized in that setting, and how does the model behave when the goal involves an unseen object state?
Reviewer_XqaUOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes EgoForge, a TI2V model specialized on egocentric scenarios. EgoForge is a Wan2.2 based model that takes an initial egocentric image, a goal prompt and optionally an exocentric view image and generates high quality egocentric dynamics. The training involves fundamentally two stages: (1) finetuning Wan2.2 on their curated egocentric dataset (X-Ego) using geometry supervision provided by VGGT features; (2) a RL refinement stage which adapts DiffusionNFT to video rollouts referenced as GRAFT. Their main contributions are:

  • a curated dataset of 15.000 training samples of egocentric videos with goal-centric text description
  • deep empirical analysis on extending DiffusionNFT (GRAFT) to video generation for egovision. Main results and extensive ablations validate the effectiveness of each additional component: exo-view, goal oriented text, geometry features and setup of GRAFT.

While I appreciate the effort put in this work, my overall judgment leans towards rejection mainly because the claimed novelties are particularly weak: the whole EgoForge model claim is quite narrow and mainly plays on the debatable definition of “goal oriented” text inputs, and the core proposed method GRAFT provides valuable empirical results but is not fundamentally different than DiffusionNFT.

Critical strengths

The paper is clear and provides a sound engineering solution for generating higher quality egocentric videos.

I’ve particularly appreciated the effort put in the many controlled experiments and ablations covering even components like VLM and the quantity of provided details covering data processing, rewards, implementation and user study setup.

While being a subset of largely used datasets, X-Ego is an appreciated contribution to the Egovision community.

Critical weaknesses

My critical points are organized by claimed contribution:

1. EgoForge

  • A. The authors themselves, in the introduction, claim a very narrow novelty about EgoForge, many works in the egocentric area, both humans (e.g. the cited Handi) and robotics[1] trained a video model from text and single image. The hand-centric instructional motion, if referred to text prompts, is a weak argument, as such previous work also uses goal oriented prompts to generate videos and derive useful demonstrations for other learning tasks.

  • B. The presented main results refer to the inference setup in which the exocentric view comes for the last frame, a non realistic input in real use cases. The results with the first frame exocentric view are presented only in the Appendix. While they are still better than the compared baselines, I found this a bit misleading the % improvements presented in the intro.

2. GRAFT

  • C. This post-training method is the core contribution of this paper. The authors empirically verify that the frame-level baseline is worse than the proposed rollout level reward, however this is not a substantially novel algorithm but rather an extension of DiffusionNFT. The use of a mulit-reward function that evaluates multiple aspects is a common approach in the literature of reward based post-training methods[2] for image/video generation. The per-condition scale is a clever idea but accounts for a small increase on overall performances.

References:

[1] Jang et al. “DREAMGEN: Unlocking Generalization in Robot Learning through Video World Models”

[2] Xue et al. “DanceGRPO: Unleashing GRPO on Visual Generation”

Questions for authors

Suggestion

  1. Since GRAFT seems to be the main contribution of this paper, I think it would be more valuable if framed as an analysis of reward-based post-training methods for egocentric video generation. So, going beyond the per-frame adaptation and internal ablations, the authors may compare with other approaches like DanceGRPO. This would help telling apart the contribution of GRAFT's optimization strategy from the proposed reward design.
  2. I would recommend either the no exocentric or first-frame exocentric view inference setup as the reference results to report. Thus changing the relative improvements presented in the introduction and main quantitive comparison of Table 2.
Reviewer_QJ7POpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • Proposes EgoForge, a goal-directed egocentric world simulator conditioned on an initial ego image, a natural-language instruction, and an optional exocentric reference.
  • Introduces VGGT-based geometry supervision to improve spatial consistency.
  • Proposes GRAFT, extending DiffusionNFT to trajectory-level reward-guided refinement for goal completion, scene consistency, temporal causality, and perceptual fidelity.
  • Introduces the X-Ego benchmark and evaluates the method through quantitative comparisons, ablations, and real-world demonstrations.
Critical strengths
  • Meaningful problem setting: EgoForge addresses goal-directed egocentric world simulation from sparse visual observations and natural-language instructions, without requiring explicit motion trajectories at inference. This is a practically relevant setting for egocentric world modeling.
  • Well-motivated GRAFT formulation with promising results: Extending reward-guided refinement to trajectory-level video generation is a meaningful adaptation for modeling procedural actions and temporal dependencies. The method demonstrates promising qualitative results and consistent quantitative improvements, supported by extensive ablation studies.
Critical weaknesses

W1. Insufficient direct evaluation of the proposed components.

  • VGGT-based supervision and exocentric conditioning aim to improve geometric consistency, while GRAFT targets goal completion and temporal causality. However, the evaluations mainly rely on conventional video-quality metrics.
  • Additional evaluations of 3D scene consistency (e.g., reprojection error) and goal completion and temporal causality (e.g., independent VLM-based evaluation) would better support these claims.
  • A controlled ablation separating VGGT supervision and exocentric conditioning would further clarify their individual contributions.

W2. Potential limitations of the exocentric-reference evaluation protocol.

  • Main quantitative evaluations use an exocentric reference from the end of the target clip, potentially revealing future scene states.
  • Although clip-start results are provided in the supplementary, using this setting as the primary evaluation would better reflect the intended future world simulation scenario.
Questions for authors

Q1. VGGT supervision

  • What is provided as input to VGGT during training? Does it supervise temporal geometry within the ego-video or cross-view geometry between Ego and Exo?

Q2. Geometry alignment design

  • What motivates separating angular and scale alignment? In particular, what additional geometric information does the scale alignment provide beyond conventional feature alignment?

Q3. GRAFT sampling efficiency

  • How does GRAFT performance vary with the number of sampled rollouts per condition (K)? What is the computational cost of the rollout generation and VLM-based reward evaluation?

Q4. Scope of goal-directed generation

  • The qualitative examples demonstrate promising procedural generation. However, can EgoForge handle more abstract goals that require implicit action planning, or unseen compositions of familiar actions?

Q5. Additional comparisons

  • Comparisons with stronger recent TI2V models, such as MiniMax-H3 or Cosmos-Predict2.5, would help contextualize EgoForge's goal-directed generation capabilities. Could the authors also clarify the exact Cosmos version used in the current experiments?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.