- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes P3-WAM, a unified world action model for egocentric dexterous manipulation that integrates three complementary capabilities: 3D hand and camera motion reconstruction (PERCEIVE), action-conditioned video prediction (PREDICT), and goal-directed joint action and video generation (PLAN). Built upon a pretrained video diffusion model, the framework introduces a structured 3D action representation and a mixture-of-transformers architecture with bidirectional video–action attention. Through randomized modality visibility, the model learns all three tasks within a unified training framework.
Experiments across four egocentric manipulation datasets demonstrate improvements over single-mode training and strong performance against specialized baselines. The authors further evaluate planning quality through VLM and human assessments, along with qualitative real-robot demonstrations.
Critical strengths
1. A well-motivated and coherent unified framework.
The paper presents an interesting approach to connecting perception, prediction, and planning within a single world action model. The integration of structured 3D hand representations with a pretrained video diffusion backbone is well motivated, and the unified training strategy provides a natural way to share knowledge across these closely related tasks.
2. Strong and comprehensive experimental results.
The experimental evaluation is convincing, with consistent improvements over single-mode training across all three capabilities. The comparisons on multiple datasets demonstrate strong performance in both hand reconstruction and action-conditioned video prediction. The additional human evaluations, backbone comparisons, and qualitative robot demonstrations further strengthen the paper.
Critical weaknesses
1. Limited analysis of cross-task contributions in unified training.
While the paper demonstrates clear benefits of unified training over single-mode training, it does not fully investigate how the three tasks contribute to one another. The current experiments mainly compare individual task training against the full unified model, making it difficult to identify which task combinations are responsible for the observed improvements.
2. Lack of quantitative evaluation of camera trajectory prediction.
Camera motion is an important component of the proposed 3D action representation and is explicitly predicted by the model. However, the quantitative evaluations primarily focus on hand reconstruction accuracy and video generation quality, without separately reporting camera trajectory estimation errors. This makes it difficult to assess how accurately the model recovers camera motion, particularly over longer sequences.
Questions for authors
Q1. Cross-task contributions in unified training.
Could the authors provide additional ablations with pairwise task combinations, such as PERCEIVE + PREDICT, PERCEIVE + PLAN, and PREDICT + PLAN, alongside the full unified model? These comparisons would help clarify how the three capabilities benefit one another and whether certain task combinations contribute more significantly to the overall improvements.
Q2. Camera trajectory prediction accuracy.
Since camera motion is explicitly modeled and predicted alongside hand trajectories, could the authors provide quantitative evaluations of camera trajectory accuracy, such as rotation and translation errors against ground truth where available? An analysis of temporal drift would also help assess the reliability of camera motion estimation over longer sequences.
I will further raise my score if the questions above are discussed and solved clearly.
3 comments
Thank you for the rebuttal and additional experiments. All of my concerns have been addressed, and I will raise my score accordingly.
Dear Reviewer At37,
Thank you so much for your follow-up and for increasing your score! We truly appreciate your time, constructive comments, and support for our work. We will ensure that all discussed details are thoroughly updated in the revised manuscript.
Best regards,
The Authors
Full response and visual evidence: Response to Reviewer At37.
We thank the reviewer for recognizing our coherent unified framework and the strong experimental results across perception, prediction, and planning. In response to your questions, we now report pairwise training ablations under a matched budget and quantitative camera evaluation covering rotation and translation accuracy, relative pose error, and temporal drift. These additions clarify cross-task contributions and demonstrate the accuracy and temporal stability of our camera predictions.
Pairwise task combinations
We compare Perceive + Plan, Predict + Plan, and Perceive + Predict under the same 5,000-update training budget as Unified and the single-mode models. The updated mode-ablation table reports Perceive and Predict results for all five settings.
Camera errors and temporal drift
We directly evaluate camera trajectories against ground truth using first-frame-relative rotation, scale-aligned camera-center error, adjacent-frame RPE, and temporal variation. Our model reduces camera-center error by 56.8%, rotation error by 26.4%, and translation RPE by 37.3% relative to VGGT. Examples A–C show raw trajectories across the full 5-second interval, with smaller final position and rotation errors.
We measure camera-position error at 2.5 and 5 seconds. Ours has lower error at both times and a smaller endpoint-minus-midpoint increase: 0.073 cm, compared with 0.194 cm for VGGT.