- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
PointWAM is a framework for dexterous robotic manipulation that jointly predicts future scene and hand motion as 3D point trajectories in a shared coordinate frame. Given the current point cloud, hand keypoints, and a language instruction, the model predicts these trajectories, while a learned retargeter converts hand trajectories and robot state into actions. It is pretrained on human demonstration videos and fine-tuned on robot demonstrations. Evaluations cover DexJoCo, RoboDojo-Precision, and two real-robot tasks. On DexJoCo, PointWAM achieves a reported average multi-task success rate of 69.0%, compared with 57.3% for the strongest baseline. Ablations examine scene-trajectory supervision, human-video pretraining, and representation choices.
Critical strengths
- A useful unified representation. Representing scene and hand motion as 3D point trajectories in a shared coordinate frame supports human-video pretraining and robot adaptation through a learned action retargeter.
- Strong DexJoCo results and supportive ablations. Multi-task success reaches 69.0%, versus 57.3% for the strongest baseline. In a matched ablation using the smaller model, scene-trajectory supervision improves success from 53.5% to 64.4%.
Critical weaknesses
- External validation remains preliminary. The strongest evidence comes from DexJoCo. Low success rates on RoboDojo-Precision and real-robot evaluation on only two tasks limit the evidence for broader applicability.
- Conceptual novelty is relatively incremental. Prior work has explored 3D hand representations, trajectory learning from human videos, and future-motion supervision. The contribution mainly lies in unified scene and dexterous-hand trajectory prediction and its empirical effectiveness.
Questions for authors
- What primarily causes the low success rate on RoboDojo-Precision, and what limitations in transfer across embodiments or tasks do these failures reveal?
- In the supplementary video, PointWAM shows abrupt motion changes around 11, 28, 42, and 57 seconds, whereas the corresponding π0.5 segments appear smoother. Were these PointWAM rollouts edited or any frames removed? Please explain these changes and whether the original continuous timing was preserved.
4 comments
Response to reviewer c9gi [2/2]
(2) Conceptual novelty (W2)
Prior work has indeed explored 3D hand representations, trajectory learning from human videos, and future-motion supervision, but PointWAM cannot be built by simply combining these ideas. As the table shows, each prior method models only a part of the scene and the hands, and none is 3D from its input through both forecasts.
| Prior work | Input features | Future scene motion | Future hand motion | Robot actions |
|---|---|---|---|---|
| Hand trajectories from human videos: Point Policy (Haldar & Pinto, 2025) | 3D object points annotated by a human | Not forecast | 3D trajectories of two gripper points | Gripper pose computed from the points |
| Hand trajectories from human videos: Dexterous Point Policy (Kim et al., 2026a) | 3D object points selected by external models | Not forecast | 3D trajectories of six hand keypoints | Inverse kinematics |
| Future-motion supervision: PointWorld (Huang et al., 2026) | 2D image features (DINOv3) projected onto 3D points | 3D trajectories of all scene points | Not forecast; robot motion is an input | Separate planner |
| Future-motion supervision: EgoWAM (Li et al., 2026a) | 2D image features (ResNet-18 or DINO) | 3D flow as a training target only, not predicted at test time | Gripper pose, no finger motion | Gripper actions |
| PointWAM (ours) | 3D features from a 3D encoder for every scene point | 3D trajectories of all scene points | 3D trajectories of ten keypoints per hand | Learned retargeter, finger-level |
The methods that learn hand trajectories from human videos receive the scene as pre-selected points and do not forecast it, and the methods with future-motion supervision forecast the scene but not a multi-fingered hand. The unified prediction of scene and dexterous-hand trajectories, which the reviewer identifies as our contribution, resolves both. It is a new formulation, not an incremental step, and it changes what each of the three ideas can do.
- A unified 3D world model of the scene and dexterous hands. The hand keypoints and every scene point are forecast by one backbone as 3D trajectories in a shared space-time coordinate frame, so the 3D hand representation extends to the whole scene and no object or keypoint has to be selected. Human videos contain both kinds of trajectories, so they supervise the whole model without robot actions, not the hand alone, at the scale of 1.15M episodes. Accordingly, pre-training on EgoDex alone raises the success rate by 48.0 pt, against at most 1.4 pt for the baselines (Fig. 6), and replacing the 3D space with image features lowers it from 64.4% to 59.0% (Table 4 (a)).
- A 3D world action model built on this world model. We pre-train the forecaster on human videos and then fine-tune it with an action retargeter, which converts the forecast hand trajectories into finger-level robot actions. Future-motion supervision thereby shapes the actions: removing the scene forecast lowers the success rate from 64.4% to 53.5%, although the retargeter never reads it, and forecasting image latents instead reaches only 60.1% (Table 4 (b)).
The contribution is therefore conceptual as well as empirical. To our knowledge, PointWAM is the first world action model for dexterous hands that perceives, forecasts, and acts in 3D.
(3) Abrupt motion in the supplementary video (Q2)
No PointWAM rollout was edited and no frames were removed. Every clip plays at 1× speed and at its recorded 30 fps, and the four moments that the reviewer noted (11, 28, 42, and 57 s) each fall inside one continuous trial, not at a cut between clips. The only cut removes the last 1.6 s of a failed GR00T N1.6 rollout, after its outcome, where a person's arm enters the frame.
The abrupt changes come from how the robot executes actions, not from the video. It executes each predicted action chunk for 1 s without temporal smoothing, so the motion can jump when the next chunk begins. Accordingly, the jumps recur at exact 1 s intervals (e.g., at 9.5, 10.5, 11.5, and 12.5 s), which removed frames would not produce. All methods run under this setting, and a jump shows only when consecutive chunks disagree, so π0.5 shows the same pattern in its square-peg rollouts (101–116 s) although it appears smoother in the banana rollouts. We added this execution setting to Appendix C.4 of the revision.
Thank you for the detailed response. The additional Bench2Dex and real-robot evaluations strengthen the evidence for dexterous manipulation, and the RoboDojo-Precision analysis clarifies the limits of transfer to parallel-jaw grippers. I also appreciate the comparison with prior work and the explanation that the abrupt movements in the video occur at action chunk boundaries without temporal smoothing. I maintain my rating of 3 (Weak acceptance).
We appreciate your careful reading of our response. We are glad the additional evaluations addressed your concerns and will include them in the final version.
Response to reviewer c9gi [1/2]
Dear reviewer c9gi, we thank you for recognizing our unified representation, the strong DexJoCo results, and the supportive ablations. Please refer to the general response for a summary of the revision. Here, we address your specific concerns.
(1) External validation and the limits of transfer (W1 & Q1)
We added two evaluations, the Bench2Dex benchmark in simulation and a second real-robot platform. The table lists them with the evaluations of the paper.
Each row reports the average success rate over its tasks.
On dexterous hands the results are consistent. PointWAM leads in all four evaluations, which span 26 tasks and ten hands, and on Bench2Dex pre-training on human videos raises its success rate on all six hands tested, from 19.3% to 56.3% on average (Table 10).
RoboDojo-Precision is the only evaluation with parallel-jaw grippers, and every method stays low there. PointWAM is still the best, and its stage-wise progress shows where success is lost.
These failures point to the pre-training data, not to the method. PointWAM is pre-trained only on human hands, whose motion carries over to multi-fingered hands and to placement, but not to grasping a small object between two jaws. GR00T N1.6 is pre-trained on gripper data and lifts these objects more often, so we expect that adding such data, e.g., DROID (Khazatsky et al., 2024), would close this gap. We leave this to future work, since our focus is dexterous manipulation. We added these results to Sec. 4.1 and 4.2 and the failure analysis to Appendix D.5 of the revision.