acceptodds
Under review as a conference paper at ICLR 2027

EgoWAM: Geometry-Grounded World Action Models for Egocentric Interaction

Abstract

World models are typically learned in image-space state representations, such as pixels or 2D features. In egocentric hand-object interaction, however, such view-dependent states are heavily perturbed by arbitrary head motion, causing learned dynamics to entangle physical interactions with incidental viewpoint shifts. We present EgoWAM, a world action model that learns geometry-grounded dynamics in a 3D semantic field. This field represents the world by anchoring dense semantic features to scene geometry to reduce the influence of viewpoint changes. EgoWAM autoregressively predicts future 3D semantic fields and hand motions, with a diffusion-based action head that models uncertainty in future hand motions and provides a context-aware prior for sampling candidate actions during planning. We then score candidate actions using a spatial-semantic optimal transport objective, which establishes soft correspondences between predicted and goal states by aligning semantic features under a 3D transport cost. Experiments on both rigid and articulated egocentric hand-object interaction datasets show improved performance in world-state prediction, motion forecasting, and goal-directed planning.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Egocentric Vision, World Models, Hand-Object Interaction, Optimal Transport

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_CFSiOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents EgoWAM for joint prediction of hand motion and scene evolution in egocentric hand-object interaction. It represents world states using 3D pointmaps with dense semantic features and autoregressively predicts MANO motion and future world states. A diffusion action prior and a spatial-semantic optimal transport objective support goal-directed trajectory search. Experiments on HOT3D and ARCTIC evaluate state prediction, hand-motion forecasting, and planning components.

Critical strengths
  1. Separating interaction dynamics from incidental camera motion is an important problem in egocentric modeling. Anchoring semantic features in a shared 3D coordinate frame and comparing states through spatial-semantic transport form a coherent approach to this problem.

  2. The combination of hand-scene interaction features, autoregressive state prediction, and a diffusion action prior connects motion generation with its predicted environmental consequences. Results on both rigid and articulated interaction datasets, together with geometry and semantic ablations, support the usefulness of the representation for offline prediction.

Critical weaknesses
  1. Applicability with estimated geometry remains unverified. Although Figure 2 depicts VGGT-based geometry extraction, Section 4.1 describes rendering world-coordinate pointmaps from annotated object models, poses, and camera poses. Evaluation with estimated geometry is not reported.

  2. The planning claims are less well supported than the prediction results. Table 3 reports planning-related state and motion errors, but no closed-loop interaction experiment is reported to verify that executing the selected actions produces the intended object-state changes.

Questions for authors
  1. Are annotation-rendered pointmaps used as model inputs as well as supervision during evaluation? How does performance change when using VGGT-estimated input geometry?

  2. Section 4.2 uses directly predicted actions for world-action models, while Table 3 evaluates planning variants. Does Table 3 involve the execution and replanning described in Section 3.3? If so, what produces the new world state after each action?

  3. Anonymized videos of predicted scene evolution and hand motions, including planning examples, would complement Figures 1 and 3 and help assess generation quality. The supplementary material currently contains only evaluation sample metadata.

7 comments
AuthorsOpenReview ↗

W1 / Q1. Pointmap inputs and supervision

Thank you for the question. Yes. In the original experiments, we rendered pointmaps from annotated object meshes, object poses, and camera poses. The four historical pointmaps served as inputs. Future pointmaps supplied targets during training and the reference for scoring during evaluation.

To test the geometry pipeline in Figure 2, we replaced only the historical pointmaps with estimates from VGGT-Ω depth. We kept EgoWAM frozen and retained the annotated camera poses. Hand MPJPE rose by less than 0.6 mm and DINO feature MSE by less than 1% on both datasets. The largest effect was on scene-point error, which rose by 3.44 mm on HOT3D and 6.23 mm on ARCTIC. We consider these increases acceptable on both datasets and find that EgoWAM is robust to replacing ground-truth input depth with estimated depth in this setting.

The table below gives the VGGT-Ω results and relative changes from rendered inputs (Section 4.3, Table 5).

DatasetHand MPJPE (mm) ↓Root translation error (mm) ↓DINO MSE ↓DINO cosine ↑Scene-point error (mm) ↓
HOT3D19.2188 (+3.15%)14.9610 (+3.08%)0.00973 (+0.78%)0.925209 (−0.06%)29.5281 (+13.19%)
ARCTIC28.8899 (+1.87%)23.2175 (+1.90%)0.00955 (+0.66%)0.928974 (−0.05%)38.4189 (+19.37%)
AuthorsOpenReview ↗

W2. Object motion after execution

Thank you for the suggestion. We tested whether executed actions move a grasped object in a MuJoCo reconstruction of one randomly selected HOT3D recording. In Move, every method took control with the can already in hand. Success required moving it to a target roughly 18–19 cm away and retaining the grasp. MuJoCo simulated contact and returned fresh observations after each action chunk.

EgoWAM was the only method to carry the can within 5 cm of its target while retaining its grasp (1/20 trials). At 8 cm, EgoWAM succeeded in 3/20 trials, UVA in 1/20, and the other methods in 0/20. The can frequently slipped during transport.

The table below gives successes over the same 20 seeds for both Move and the bottle-reaching task (Section 4.4, Table 6). Appendix A.5, Figure 4 shows the resulting hand and object positions.

MethodReach ≤3 cmReach ≤4 cmMove ≤5 cmMove ≤8 cm
EgoWAM10/2014/201/203/20
UVA6/209/200/201/20
Fast-WAM1/201/200/200/20
V-JEPA2-AC0/200/200/200/20
DINO-WM0/200/200/200/20
NWM0/200/200/200/20
AuthorsOpenReview ↗

Q2. Offline search and closed-loop execution

Thank you for the question. The submitted Table 3 evaluates offline search from a fixed observation and goal (Section 4.3, Table 4 in the revision). EgoWAM predicts each candidate’s outcome, and its distance to the goal determines which trajectories are refined in the next round. The search ends with trajectory selection, before any action is executed or any new observation is received.

In the separate closed-loop experiment, MuJoCo supplies the next observation after executing each action sequence. Please see our response to W2 above for the setup and results (Section 4.4, Table 6).

Q3. Visualizations and supplementary videos

Thank you for suggesting videos. For now, please see the new visualizations of Reach and Move in Figure 4 (Appendix A.5). They show the common starting state and the final hand and object positions for Fast-WAM, UVA, and EgoWAM alongside the recorded motion.

We will include anonymized videos of the full planning process and simulator executions in a future version of the supplementary material.

Reviewer_Cq5tOpenReview ↗
Rating
2(+1)
Confidence
1(−2)
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies world-action modeling for egocentric hand-object interaction. Its core motivation is that changes in images and 2D features reflect both physical interactions and camera motion, which can become entangled when dynamics are modeled directly in these spaces. The authors anchor semantic features to 3D pointmaps in a common world coordinate frame to construct a 3D semantic field. Hand states are represented using MANO, and an interaction field is constructed from the distances and directions of scene points near the hand surface.

Critical strengths

Anchoring semantics to geometric locations in a common coordinate frame and comparing goal states in this representation directly addresses camera-motion interference in egocentric scenarios. This idea has research value for egocentric prediction, interaction understanding, and manipulation-oriented state representations.

Critical weaknesses
  1. The choice of baselines is inappropriate. Hand motion forecasting is a well-established research area, yet none of the baselines are hand motion forecasting methods. Instead, all are world models or world-action models for robotics. Relevant methods such as Uni-Hand: Universal Hand Motion Forecasting in Egocentric Views should be included.

  2. Action-conditioned world models receive observation history and a future goal, and search for actions using CEM, whereas world-action models directly output predicted actions. Both are ultimately evaluated against the same recorded human trajectory. This protocol compares different task formulations and search difficulties using the same metrics.

Questions for authors
  1. I would like the authors to address my concern about the baseline comparisons in their response, and I would also like to discuss this issue with the other reviewers. This is the primary reason for my score of 1. If this issue is resolved, I will raise my score to 2 or 3.

  2. How do world-state predictions change when actions are perturbed or removed?

  3. Could the authors provide multi-step rollout results and independent validation of the outcomes of different candidate actions?

7 comments
AuthorsOpenReview ↗

W1 / Q1. Full-hand comparison with Uni-Hand

Thank you for recommending Uni-Hand. On each dataset, we trained its wrist-only model and a 21-joint adaptation from the same released checkpoint. The adaptation takes the history of all 21 joints per hand and predicts their future positions. We omitted the contact-event head and its loss because HOT3D and ARCTIC lack compatible event labels (Appendix A.4).

On the same 21-joint task, EgoWAM has lower mean per-joint position error (MPJPE) than Uni-Hand on both datasets. The errors are 18.632 vs. 65.685 mm on HOT3D and 28.360 vs. 62.067 mm on ARCTIC. On its original wrist-only task, Uni-Hand obtains wrist errors of 44.087 and 16.384 mm, respectively.

The table below compares Uni-Hand and EgoWAM on the full-hand task (Section 4.2, Table 1 of the revised manuscript).

DatasetMethodMPJPE ↓MPJVE ↓F@15 ↑RTE ↓
HOT3DUni-Hand65.685177.2830.14996.779
HOT3DEgoWAM18.6327.3790.83614.514
ARCTICUni-Hand62.067227.8560.23755.463
ARCTICEgoWAM28.3608.9260.71222.785
AuthorsOpenReview ↗

W2. Action search and task success

Thank you for raising both concerns. In Section 4.2, Table 1, each method uses its own procedure to produce hand actions. The hand-motion columns therefore compare complete systems on how closely their actions match the recorded expert trajectory. We agree that the methods face different search difficulties and that a trajectory can reach the goal even if it differs from the expert’s. Trajectory error alone cannot establish planning success. We added two experiments to address these points.

Shared action candidates. To control for differences in proposal generation, we gave DINO-WM, V-JEPA2-AC, NWM, and EgoWAM the same history, goal, and 64 four-frame trajectories. We sampled 32 trajectories from the frozen EgoWAM prior and generated 32 by perturbing the recorded motion, excluding the unperturbed demonstration. Each model ranked the candidates once. Relative to their Gaussian CEM results in Table 1, all three baselines reduced mean per-joint position error by at least 40% on each dataset. Even with these gains, EgoWAM ranked first on all four motion metrics on both datasets.

The table below gives the shared-candidate results (Section 4.2, Table 2).

DatasetMethodMPJPE ↓MPJVE ↓F@15 ↑RTE ↓
HOT3DDINO-WM73.688106.9230.20765.022
HOT3DV-JEPA2-AC39.91550.1640.46633.926
HOT3DNWM40.97658.0770.07232.027
HOT3DEgoWAM18.6327.3790.83614.514
ARCTICDINO-WM77.29292.3250.06367.472
ARCTICV-JEPA2-AC32.81534.3430.23226.277
ARCTICNWM39.25254.2350.05231.187
ARCTICEgoWAM28.3608.9260.71222.785

Closed-loop task success. We executed each method’s actions in a MuJoCo reconstruction of one randomly selected HOT3D recording. All six methods started from the same state for each task. We scored success from the simulated hand and object positions, so any trajectory that reached the target and met the task criteria counted. EgoWAM reached within 4 cm of the bottle in 14/20 trials, compared with 9/20 for UVA. It carried the can to within 8 cm of the target while retaining its grasp in 3/20 trials, compared with 1/20 for UVA and 0/20 for the other methods. EgoWAM achieved the highest success count at all four tolerances, although successful transport remained infrequent.

The table below gives successes over the same 20 seeds per task (Section 4.4, Table 6). Appendix A.5 details the setup and scoring.

MethodReach ≤3 cmReach ≤4 cmMove ≤5 cmMove ≤8 cm
EgoWAM10/2014/201/203/20
UVA6/209/200/201/20
Fast-WAM1/201/200/200/20
V-JEPA2-AC0/200/200/200/20
DINO-WM0/200/200/200/20
NWM0/200/200/200/20
AuthorsOpenReview ↗

Q2. Scene response to hand motion

Thank you for the suggestion. On HOT3D examples with a stable grasp, we held the visual history fixed and varied only the hand action. We first froze the hand by setting all motion increments to zero, then shifted its endpoint along one axis from −15 to +15 mm in 1 mm steps.

Every nonzero offset produced a larger predicted object-centroid displacement than the frozen-hand control. For example, the centroid moved approximately 0 mm with the hand fixed, 1.42 mm for a −4 mm offset, and 0.48 mm for a +4 mm offset. The response varied with the action but did not increase monotonically with offset magnitude. Appendix A.6, Figure 5 shows the full sweep.

Reviewer_ZFcJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EgoWAM predicts hand actions and scene states in a 3D semantic field. A diffusion action prior and spatial-semantic optimal transport objective guide candidate trajectory search.

Critical strengths

Place the interactive state and target distance in the same three-dimensional coordinate system, with a clear task motivation. Evaluation covers both rigid and articulated interactions.

Critical weaknesses
  1. Action-conditioned baselines search MANO space using Gaussian CEM, whereas world-action models generate actions directly. These different proposal distributions confound the motion comparison; evaluate world models with shared action candidates or a common learned prior.

  2. Experiments use pointmaps rendered from ground-truth CAD models and object/camera poses; the illustrated VGGT pipeline is not evaluated. The advantage is therefore confounded with privileged geometry.

  3. Offline trajectory error is not yet sufficient to prove closed-loop target control. This paper evaluates planning with the goal cost of predicted states, using recorded actions as a reference on pre-recorded human interaction data, but does not provide the actual state changes or target success rates produced by candidate actions after execution in an interactive environment.

Questions for authors

See weaknesses.

3 comments
AuthorsOpenReview ↗

W1. Shared action candidates across baselines

Thank you for the suggestion. For each test example, we gave all four models the same history, goal, and 64 four-frame bimanual trajectories. We generated 32 trajectories by denoising Gaussian noise with the frozen EgoWAM prior and 32 by perturbing the recorded motion, excluding the unperturbed demonstration. Each model ranked all 64 candidates once by predicted goal discrepancy in its own state space. We compared its top-ranked trajectory with the recorded motion.

With shared candidates, all three baselines reduce their mean per-joint position error (MPJPE) by at least 40% on each dataset relative to their Gaussian CEM results in Table 1 (“Hand Motion (Native Protocol)”). Even with these gains, EgoWAM ranks first on all four hand-motion metrics on both HOT3D and ARCTIC.

The table below shows the shared-candidate results (Section 4.2, Table 2 of the revised manuscript).

DatasetMethodMPJPE ↓MPJVE ↓F@15 ↑RTE ↓
HOT3DDINO-WM73.688106.9230.20765.022
HOT3DV-JEPA2-AC39.91550.1640.46633.926
HOT3DNWM40.97658.0770.07232.027
HOT3DEgoWAM18.6327.3790.83614.514
ARCTICDINO-WM77.29292.3250.06367.472
ARCTICV-JEPA2-AC32.81534.3430.23226.277
ARCTICNWM39.25254.2350.05231.187
ARCTICEgoWAM28.3608.9260.71222.785
AuthorsOpenReview ↗

W2. Robustness to estimated geometry

Thank you for the suggestion. We originally used ground-truth geometry to isolate forecasting error from errors in the input depth and measure how accurately EgoWAM predicts from a known initial scene. To test robustness to depth estimation errors, we froze EgoWAM and replaced the four historical pointmaps with pointmaps reconstructed from VGGT-Ω depth. We kept the annotated camera poses, test examples, semantic features, hand inputs, and evaluation targets fixed.

With estimated depth, hand MPJPE increases by less than 0.6 mm and DINO feature MSE by less than 1% on both datasets. Scene geometry is more sensitive. Errors in the input pointmaps can propagate into the predicted 3D positions and compound the model’s forecasting error. Even so, scene-point error increases by only 3.44 mm on HOT3D and 6.23 mm on ARCTIC. We consider these increases acceptable on both datasets and find that EgoWAM is robust to replacing ground-truth input depth with estimated depth in this setting.

The table below gives the VGGT-Ω results and relative changes from rendered inputs (Section 4.3, Table 5).

DatasetHand MPJPE (mm) ↓Root translation error (mm) ↓DINO MSE ↓DINO cosine ↑Scene-point error (mm) ↓
HOT3D19.2188 (+3.15%)14.9610 (+3.08%)0.00973 (+0.78%)0.925209 (−0.06%)29.5281 (+13.19%)
ARCTIC28.8899 (+1.87%)23.2175 (+1.90%)0.00955 (+0.66%)0.928974 (−0.05%)38.4189 (+19.37%)
AuthorsOpenReview ↗

W3. Closed-loop target control

Thank you for the suggestion. We tested six methods in a MuJoCo reconstruction of one randomly selected HOT3D recording. All methods started from the same state for each task. MuJoCo executed their actions, simulated contact, and returned the next observation. Reach required approaching the bottle within one second. Move required carrying the can roughly 18–19 cm to its target and retaining the grasp. Appendix A.5 of the revised manuscript details the scene reconstruction, hand control, observation timing, and success criteria for both tasks.

EgoWAM achieved the highest success count at all four tolerances. It reached within 4 cm of the bottle in 14/20 trials, versus 9/20 for UVA, the strongest baseline. Transport was harder for every method. EgoWAM brought the can within 8 cm in 3/20 trials and was the only method to succeed at 5 cm (1/20).

The table below gives successes over the same 20 seeds per task (Section 4.4, Table 6). Appendix A.5, Figure 4 shows both tasks.

MethodReach ≤3 cmReach ≤4 cmMove ≤5 cmMove ≤8 cm
EgoWAM10/2014/201/203/20
UVA6/209/200/201/20
Fast-WAM1/201/200/200/20
V-JEPA2-AC0/200/200/200/20
DINO-WM0/200/200/200/20
NWM0/200/200/200/20

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.