- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a streaming framework that forecasts hand motion, camera motion, and hand contacts from egocentric observations. A state-conditioned decoder extracts trajectories from intermediate features of an autoregressive video model (Wan2.2-TI2V-5B), while incoming observations update the forecasts. Experiments across four datasets evaluate forecasting accuracy and key design choices, with qualitative demonstrations of continuous prediction and contact-guided interaction in physics simulation.
Critical strengths
- Useful structured forecasting formulation. The method jointly predicts articulated hand motion, camera motion, and hand–object proximity, providing explicit interaction cues that are more directly usable by downstream systems than generated video alone.
- Broad evaluation with informative ablations. Experiments across four egocentric datasets demonstrate competitive forecasting performance, and ablations clarify the contributions of visual history, state conditioning, wrist anchoring, and denoising readout choices.
Critical weaknesses
- It is unclear whether the improvement comes from the video generation model’s ability. The main experiments use features from the first denoising call, before the future latent has been updated. Adapting the video generator to egocentric videos barely changes forecasting accuracy (Table 8). The key issue is that the results do not show whether these priors drive the gains, or whether a strong feature extractor and a decoder trained with extensive geometric supervision are sufficient. A comparison with a non-generative video encoder under the same training setup would help answer this question.
- A direct generate-then-reconstruct baseline is missing. The paper does not compare intermediate-feature decoding against generating future RGB videos with the same backbone and then applying hand reconstruction and contact estimation, similar to work in [1]. Such a comparison is needed to show the proposed design’s advantages in accuracy, efficiency, and temporal consistency.
- Robustness to estimated current states remains untested. Quantitative experiments use annotated current states for forecasting; even comparisons without hand-state input align predictions to the GT current wrist during evaluation. Wrist rebasing alone reduces average error from 9.33 to 7.82 cm (Table 3), showing heavy reliance on accurate localization. Evaluation with estimated hand and camera states is needed to evaluate robustness under realistic input noises.
- Lacking comparison with SOTA method and incomplete metrics: How Do I Do That [2] also works on forecasting of 3D hand motion and contacts, which is not compared against. Also, the metrics on contact state are hand-level, lacking finger-level precision & recall, or F1 score.
[1] Large Video Planner, CoRL 2025 [2] How Do I Do That? Synthesizing 3D Hand Motion and Contacts for Everyday Interactions, CVPR 2025
Questions for authors
- Does the improvement come from the video generation model’s ability or extensive geometric supervision?
- Can you compare it with a baseline that generates first and then performs reconstruction? Generally, using internal features to learn will be better than a two-stage approach, but a comparison is needed to show the effectiveness of network designs. Also, comparison with related SOTA How Do I Do That is needed.
- Robustness to estimated current states?