- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents EgoWAM for joint prediction of hand motion and scene evolution in egocentric hand-object interaction. It represents world states using 3D pointmaps with dense semantic features and autoregressively predicts MANO motion and future world states. A diffusion action prior and a spatial-semantic optimal transport objective support goal-directed trajectory search. Experiments on HOT3D and ARCTIC evaluate state prediction, hand-motion forecasting, and planning components.
Critical strengths
-
Separating interaction dynamics from incidental camera motion is an important problem in egocentric modeling. Anchoring semantic features in a shared 3D coordinate frame and comparing states through spatial-semantic transport form a coherent approach to this problem.
-
The combination of hand-scene interaction features, autoregressive state prediction, and a diffusion action prior connects motion generation with its predicted environmental consequences. Results on both rigid and articulated interaction datasets, together with geometry and semantic ablations, support the usefulness of the representation for offline prediction.
Critical weaknesses
-
Applicability with estimated geometry remains unverified. Although Figure 2 depicts VGGT-based geometry extraction, Section 4.1 describes rendering world-coordinate pointmaps from annotated object models, poses, and camera poses. Evaluation with estimated geometry is not reported.
-
The planning claims are less well supported than the prediction results. Table 3 reports planning-related state and motion errors, but no closed-loop interaction experiment is reported to verify that executing the selected actions produces the intended object-state changes.
Questions for authors
-
Are annotation-rendered pointmaps used as model inputs as well as supervision during evaluation? How does performance change when using VGGT-estimated input geometry?
-
Section 4.2 uses directly predicted actions for world-action models, while Table 3 evaluates planning variants. Does Table 3 involve the execution and replanning described in Section 3.3? If so, what produces the new world state after each action?
-
Anonymized videos of predicted scene evolution and hand motions, including planning examples, would complement Figures 1 and 3 and help assess generation quality. The supplementary material currently contains only evaluation sample metadata.
7 comments
W2. Object motion after execution
Thank you for the suggestion. We tested whether executed actions move a grasped object in a MuJoCo reconstruction of one randomly selected HOT3D recording. In Move, every method took control with the can already in hand. Success required moving it to a target roughly 18–19 cm away and retaining the grasp. MuJoCo simulated contact and returned fresh observations after each action chunk.
EgoWAM was the only method to carry the can within 5 cm of its target while retaining its grasp (1/20 trials). At 8 cm, EgoWAM succeeded in 3/20 trials, UVA in 1/20, and the other methods in 0/20. The can frequently slipped during transport.
The table below gives successes over the same 20 seeds for both Move and the bottle-reaching task (Section 4.4, Table 6). Appendix A.5, Figure 4 shows the resulting hand and object positions.
| Method | Reach ≤3 cm | Reach ≤4 cm | Move ≤5 cm | Move ≤8 cm |
|---|---|---|---|---|
| EgoWAM | 10/20 | 14/20 | 1/20 | 3/20 |
| UVA | 6/20 | 9/20 | 0/20 | 1/20 |
| Fast-WAM | 1/20 | 1/20 | 0/20 | 0/20 |
| V-JEPA2-AC | 0/20 | 0/20 | 0/20 | 0/20 |
| DINO-WM | 0/20 | 0/20 | 0/20 | 0/20 |
| NWM | 0/20 | 0/20 | 0/20 | 0/20 |
Q2. Offline search and closed-loop execution
Thank you for the question. The submitted Table 3 evaluates offline search from a fixed observation and goal (Section 4.3, Table 4 in the revision). EgoWAM predicts each candidate’s outcome, and its distance to the goal determines which trajectories are refined in the next round. The search ends with trajectory selection, before any action is executed or any new observation is received.
In the separate closed-loop experiment, MuJoCo supplies the next observation after executing each action sequence. Please see our response to W2 above for the setup and results (Section 4.4, Table 6).
Q3. Visualizations and supplementary videos
Thank you for suggesting videos. For now, please see the new visualizations of Reach and Move in Figure 4 (Appendix A.5). They show the common starting state and the final hand and object positions for Fast-WAM, UVA, and EgoWAM alongside the recorded motion.
We will include anonymized videos of the full planning process and simulator executions in a future version of the supplementary material.
W1 / Q1. Pointmap inputs and supervision
Thank you for the question. Yes. In the original experiments, we rendered pointmaps from annotated object meshes, object poses, and camera poses. The four historical pointmaps served as inputs. Future pointmaps supplied targets during training and the reference for scoring during evaluation.
To test the geometry pipeline in Figure 2, we replaced only the historical pointmaps with estimates from VGGT-Ω depth. We kept EgoWAM frozen and retained the annotated camera poses. Hand MPJPE rose by less than 0.6 mm and DINO feature MSE by less than 1% on both datasets. The largest effect was on scene-point error, which rose by 3.44 mm on HOT3D and 6.23 mm on ARCTIC. We consider these increases acceptable on both datasets and find that EgoWAM is robust to replacing ground-truth input depth with estimated depth in this setting.
The table below gives the VGGT-Ω results and relative changes from rendered inputs (Section 4.3, Table 5).