acceptodds
Under review as a conference paper at ICLR 2027

PointWAM: 3D World Action Modeling for Dexterous Robotic Manipulation

Abstract

World action models jointly learn to forecast world dynamics and predict robot actions, such that the learned internal world dynamics guide accurate actions. Existing approaches typically represent the world as RGB frames or latent counterparts while predicting actions as end-effector poses or joint angles, but they often struggle to capture the 3D spatial structure and contact geometry central to dexterous manipulation. We introduce Point World Action Model (PointWAM), a 3D world action model that decomposes the world into a scene (*i.e.*, environment) and hands (*i.e.*, actor), and jointly forecasts both as 3D point trajectories within a shared space-time coordinate frame. This explicit, disentangled representation enables effective pre-training on large-scale human demonstration videos without requiring any task-specific object or keypoint selection. Given a colored point cloud and a language instruction, PointWAM predicts how the scene and hands co-evolve in 3D space over time, then retargets the forecast hand motion to robot actions. On ten DexJoCo tasks, PointWAM surpasses the prior state-of-the-art average multi-task success rate by 11.7 percentage points, with scene-trajectory supervision contributing 10.9 points and human-data pre-training 56.9 points. On a real robot, PointWAM also outperforms GR00T N1.6 and on both single-arm and bimanual tasks, by up to 25 percentage points.

Primary area:
applications to robotics, autonomy, planning
Keywords:
dexterous robotic manipulation, world action models, vision-language-action models, point policy

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_c9giOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PointWAM is a framework for dexterous robotic manipulation that jointly predicts future scene and hand motion as 3D point trajectories in a shared coordinate frame. Given the current point cloud, hand keypoints, and a language instruction, the model predicts these trajectories, while a learned retargeter converts hand trajectories and robot state into actions. It is pretrained on human demonstration videos and fine-tuned on robot demonstrations. Evaluations cover DexJoCo, RoboDojo-Precision, and two real-robot tasks. On DexJoCo, PointWAM achieves a reported average multi-task success rate of 69.0%, compared with 57.3% for the strongest baseline. Ablations examine scene-trajectory supervision, human-video pretraining, and representation choices.

Critical strengths
  1. A useful unified representation. Representing scene and hand motion as 3D point trajectories in a shared coordinate frame supports human-video pretraining and robot adaptation through a learned action retargeter.
  2. Strong DexJoCo results and supportive ablations. Multi-task success reaches 69.0%, versus 57.3% for the strongest baseline. In a matched ablation using the smaller model, scene-trajectory supervision improves success from 53.5% to 64.4%.
Critical weaknesses
  1. External validation remains preliminary. The strongest evidence comes from DexJoCo. Low success rates on RoboDojo-Precision and real-robot evaluation on only two tasks limit the evidence for broader applicability.
  2. Conceptual novelty is relatively incremental. Prior work has explored 3D hand representations, trajectory learning from human videos, and future-motion supervision. The contribution mainly lies in unified scene and dexterous-hand trajectory prediction and its empirical effectiveness.
Questions for authors
  1. What primarily causes the low success rate on RoboDojo-Precision, and what limitations in transfer across embodiments or tasks do these failures reveal?
  2. In the supplementary video, PointWAM shows abrupt motion changes around 11, 28, 42, and 57 seconds, whereas the corresponding π0.5 segments appear smoother. Were these PointWAM rollouts edited or any frames removed? Please explain these changes and whether the original continuous timing was preserved.
4 comments
AuthorsOpenReview ↗

Response to reviewer c9gi [1/2]

Dear reviewer c9gi, we thank you for recognizing our unified representation, the strong DexJoCo results, and the supportive ablations. Please refer to the general response for a summary of the revision. Here, we address your specific concerns.

(1) External validation and the limits of transfer (W1 & Q1)

We added two evaluations, the Bench2Dex benchmark in simulation and a second real-robot platform. The table lists them with the evaluations of the paper.

EvaluationRobotTasksStrongest baseline SR (%) ↑PointWAM SR (%) ↑
DexJoCoFranka + Allegro1057.3 (GR00T N1.6)69.0
Bench2Dex (new)seven dexterous hands1335.5 (GR00T N1.5)55.2
Real robotOpenArm + Inspire RH56F1241.7 (π0.5)60.4
Real robot (new)RB-Y1 + Wuji125.0 (π0.5)41.7
RoboDojo-PrecisionARX X5 + parallel-jaw grippers82.3 (GR00T N1.6)4.8

Each row reports the average success rate over its tasks.

On dexterous hands the results are consistent. PointWAM leads in all four evaluations, which span 26 tasks and ten hands, and on Bench2Dex pre-training on human videos raises its success rate on all six hands tested, from 19.3% to 56.3% on average (Table 10).

RoboDojo-Precision is the only evaluation with parallel-jaw grippers, and every method stays low there. PointWAM is still the best, and its stage-wise progress shows where success is lost.

  • Placement transfers. PointWAM inserts at least one tube in 80% of episodes, against 52% for GR00T N1.6, and pre-training raises its insert-tubes score from 15.2 to 32.4.
  • Small-object grasping does not. PointWAM fails to lift the key and the coin in 80% and 76% of episodes, against 48% and 70% for GR00T N1.6, and pre-training on human videos brings no clear gain on these tasks.
  • Millimeter-level alignment fails for every method. Neither PointWAM nor GR00T N1.6 inserts a lifted key, and no method completes fasten screws, which requires seating a nut within 1 mm.

These failures point to the pre-training data, not to the method. PointWAM is pre-trained only on human hands, whose motion carries over to multi-fingered hands and to placement, but not to grasping a small object between two jaws. GR00T N1.6 is pre-trained on gripper data and lifts these objects more often, so we expect that adding such data, e.g., DROID (Khazatsky et al., 2024), would close this gap. We leave this to future work, since our focus is dexterous manipulation. We added these results to Sec. 4.1 and 4.2 and the failure analysis to Appendix D.5 of the revision.

AuthorsOpenReview ↗

Response to reviewer c9gi [2/2]

(2) Conceptual novelty (W2)

Prior work has indeed explored 3D hand representations, trajectory learning from human videos, and future-motion supervision, but PointWAM cannot be built by simply combining these ideas. As the table shows, each prior method models only a part of the scene and the hands, and none is 3D from its input through both forecasts.

Prior workInput featuresFuture scene motionFuture hand motionRobot actions
Hand trajectories from human videos: Point Policy (Haldar & Pinto, 2025)3D object points annotated by a humanNot forecast3D trajectories of two gripper pointsGripper pose computed from the points
Hand trajectories from human videos: Dexterous Point Policy (Kim et al., 2026a)3D object points selected by external modelsNot forecast3D trajectories of six hand keypointsInverse kinematics
Future-motion supervision: PointWorld (Huang et al., 2026)2D image features (DINOv3) projected onto 3D points3D trajectories of all scene pointsNot forecast; robot motion is an inputSeparate planner
Future-motion supervision: EgoWAM (Li et al., 2026a)2D image features (ResNet-18 or DINO)3D flow as a training target only, not predicted at test timeGripper pose, no finger motionGripper actions
PointWAM (ours)3D features from a 3D encoder for every scene point3D trajectories of all scene points3D trajectories of ten keypoints per handLearned retargeter, finger-level

The methods that learn hand trajectories from human videos receive the scene as pre-selected points and do not forecast it, and the methods with future-motion supervision forecast the scene but not a multi-fingered hand. The unified prediction of scene and dexterous-hand trajectories, which the reviewer identifies as our contribution, resolves both. It is a new formulation, not an incremental step, and it changes what each of the three ideas can do.

  1. A unified 3D world model of the scene and dexterous hands. The hand keypoints and every scene point are forecast by one backbone as 3D trajectories in a shared space-time coordinate frame, so the 3D hand representation extends to the whole scene and no object or keypoint has to be selected. Human videos contain both kinds of trajectories, so they supervise the whole model without robot actions, not the hand alone, at the scale of 1.15M episodes. Accordingly, pre-training on EgoDex alone raises the success rate by 48.0 pt, against at most 1.4 pt for the baselines (Fig. 6), and replacing the 3D space with image features lowers it from 64.4% to 59.0% (Table 4 (a)).
  2. A 3D world action model built on this world model. We pre-train the forecaster on human videos and then fine-tune it with an action retargeter, which converts the forecast hand trajectories into finger-level robot actions. Future-motion supervision thereby shapes the actions: removing the scene forecast lowers the success rate from 64.4% to 53.5%, although the retargeter never reads it, and forecasting image latents instead reaches only 60.1% (Table 4 (b)).

The contribution is therefore conceptual as well as empirical. To our knowledge, PointWAM is the first world action model for dexterous hands that perceives, forecasts, and acts in 3D.

(3) Abrupt motion in the supplementary video (Q2)

No PointWAM rollout was edited and no frames were removed. Every clip plays at 1× speed and at its recorded 30 fps, and the four moments that the reviewer noted (11, 28, 42, and 57 s) each fall inside one continuous trial, not at a cut between clips. The only cut removes the last 1.6 s of a failed GR00T N1.6 rollout, after its outcome, where a person's arm enters the frame.

The abrupt changes come from how the robot executes actions, not from the video. It executes each predicted action chunk for 1 s without temporal smoothing, so the motion can jump when the next chunk begins. Accordingly, the jumps recur at exact 1 s intervals (e.g., at 9.5, 10.5, 11.5, and 12.5 s), which removed frames would not produce. All methods run under this setting, and a jump shows only when consecutive chunks disagree, so π0.5 shows the same pattern in its square-peg rollouts (101–116 s) although it appears smoother in the banana rollouts. We added this execution setting to Appendix C.4 of the revision.

Thank you for the detailed response. The additional Bench2Dex and real-robot evaluations strengthen the evidence for dexterous manipulation, and the RoboDojo-Precision analysis clarifies the limits of transfer to parallel-jaw grippers. I also appreciate the comparison with prior work and the explanation that the abrupt movements in the video occur at action chunk boundaries without temporal smoothing. I maintain my rating of 3 (Weak acceptance).

We appreciate your careful reading of our response. We are glad the additional evaluations addressed your concerns and will include them in the final version.

Reviewer_uBvZOpenReview ↗
Rating
3
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents PointWAM, which jointly predicts scene and hand trajectories in 3D, learns from human demonstrations, and retargets predicted hand motion to robot actions for dexterous manipulation.

Critical strengths
  1. The paper is clearly written and well organized, making the overall approach easy to follow.

  2. The experimental evaluation is comprehensive.

Critical weaknesses
  1. I remain unconvinced by the degree of novelty relative to prior work on 3D flow prediction and point-based policies. Although Section 2 and Appendix A discuss these connections, the contribution beyond combining existing ideas in scene-motion forecasting, hand-trajectory prediction, and action retargeting could be established more convincingly.

  2. Differences in input modalities complicate the interpretation of the baseline comparisons, particularly on the real robot, where PointWAM uses depth-derived geometry while the compared policies are RGB-based. The paper acknowledges this difference, includes PointACT in simulation, and provides a controlled scene-supervision ablation. Nevertheless, the real-world comparisons do not clearly separate the benefit of additional geometric information from that of the proposed method.

  3. The explanation for the unusually large benefit from EgoDex pre-training is not fully convincing. PointWAM improves from 12.1% to 60.1%, whereas the baselines change by at most 1.4 percentage points. The different initialization conditions acknowledged in Section 4.3 make it difficult to attribute this gap specifically to the shared 3D representation.

  4. Despite the hardware validation, the real-world evaluation remains limited to two tasks on one platform. A broader range of more complex dexterous tasks would strengthen the empirical evidence.

Questions for authors

please address the concerns above

5 comments
AuthorsOpenReview ↗

Response to reviewer uBvZ [1/3]

Dear reviewer uBvZ, we thank you for finding the paper clearly written and well organized, with a comprehensive experimental evaluation. Please refer to the general response for a summary of the revision. Here, we address your specific concerns.

(1) Novelty beyond combining existing ideas (W1)

Each part of PointWAM relates to prior work, but PointWAM cannot be built by simply combining these parts. As the table shows, each prior method models only a part of the scene and the hands, and none is 3D from its input through both forecasts. The last three columns are the three parts that the reviewer lists.

Prior workInput featuresFuture scene motionFuture hand motionRobot actions
3D flow prediction: PointWorld (Huang et al., 2026)2D image features (DINOv3) projected onto 3D points3D trajectories of all scene pointsNot forecast; robot motion is an inputSeparate planner
3D flow prediction: 3PoinTr (Hung et al., 2026)3D point coordinates3D trajectories of object pointsNot forecast; hand points are removedSeparate policy trained on robot data
3D flow prediction: EgoWAM (Li et al., 2026a)2D image features (ResNet-18 or DINO)3D flow as a training target only, not predicted at test timeGripper pose, no finger motionGripper actions
Point-based policy: Point Policy (Haldar & Pinto, 2025)3D object points annotated by a humanNot forecast3D trajectories of two gripper pointsGripper pose computed from the points
Point-based policy: Dexterous Point Policy (Kim et al., 2026a)3D object points selected by external modelsNot forecast3D trajectories of six hand keypointsInverse kinematics
PointWAM (ours)3D features from a 3D encoder for every scene point3D trajectories of all scene points3D trajectories of ten keypoints per handLearned retargeter, finger-level

The 3D flow prediction methods forecast the scene but not a multi-fingered hand, and the point-based policies forecast the hand but receive the scene as pre-selected points. To resolve both and to learn from human videos at scale, we propose a new architecture with two parts.

  1. A unified 3D world model of the scene and dexterous hands. We decompose the world into the scene (the environment) and the hands (the actor), and one backbone forecasts both as 3D point trajectories in a shared space-time coordinate frame, from 3D scene features. The hand trajectories serve as an action representation that human and robot hands share, so human videos supervise the whole forecaster without robot actions, at the scale of 1.15M episodes. This is what makes human videos effective: pre-training on EgoDex alone raises the success rate by 48.0 pt, whereas the same videos change the baselines by at most 1.4 pt (Fig. 6). The 3D design is necessary on both sides, since lifted image features as input cost 5.4 pt and image latents as the forecast target cost 4.3 pt (Table 4 (a), (b)).
  2. A 3D world action model built on this world model. We pre-train the forecaster on human videos and then fine-tune it on robot demonstrations with an action retargeter, which converts the forecast hand trajectories into finger-level robot actions. PointWAM thus needs neither a separate planner nor pre-selected object points. Moreover, the learned world dynamics guide the actions: removing the scene forecast lowers the success rate by 10.9 pt, although the retargeter reads only the hand trajectories (Table 4 (b)).

To our knowledge, PointWAM is thus the first world action model for dexterous hands that perceives, forecasts, and acts in 3D.

AuthorsOpenReview ↗

Response to reviewer uBvZ [2/3]

(2) Separating the geometric input from the proposed method (W2)

We added the two baselines that the real-robot comparison lacked. PointACT takes the same depth-derived point cloud as PointWAM, and Fast-WAM is also a world action model, but on RGB (24 trials per method and task).

TaskMetricπ0.5GR00T N1.6Fast-WAMPointACTPointWAM
Banana pick-and-placeIn-domain SR (%) ↑91.783.341.78.383.3
Banana pick-and-placeOut-of-domain SR (%) ↑25.016.78.30.058.3
Square-peg insertionSR (%) ↑25.016.70.00.050.0
Square-peg insertionScore (0–3) ↑1.921.961.460.002.33

PointACT and PointWAM use depth, and the other methods use RGB. Banana pick-and-place has 12 in-domain trials with the banana and 12 out-of-domain trials with an unseen box whose color resembles the table. The score is the number of completed stages (lifting the peg, lifting the hole, inserting the peg), averaged over the trials.

Neither the depth input nor world modeling alone explains the gain. With the same depth input, PointACT never lifts the peg, and with world modeling but without 3D, Fast-WAM lifts it in every trial but never inserts it. Both stay far below PointWAM and even below the RGB policies on both tasks.

To see why depth does not help PointACT, we tested whether each model decides where to reach from the scene or from its own robot state. We take two demonstrations with the banana at different positions, so the hand has to reach different places. We then replace one input of the first demonstration with that of the second, keep the other inputs, and measure how far the predicted reach follows the replaced input.

Replaced inputPointACT follows the replaced input (%)PointWAM follows the replaced input (%)
Point cloud1101
Robot state805

0% means the predicted wrist position does not move, so the model ignores the replaced input. 100% means it moves all the way to where the wrist goes in the demonstration that the replaced input comes from. Measured 1 s after the start of the episode and averaged over ten pairs of demonstrations.

When only the point cloud changes, PointWAM moves its reach fully to the new banana position, whereas PointACT does not move it. When only the robot state changes, the banana has not moved, so the reach should stay. PointWAM keeps it, whereas PointACT moves its reach by 80%. PointWAM thus decides where to reach from the scene, and PointACT mostly from its robot state, so it does not fully utilize its 3D input. A depth input alone is therefore not sufficient, and the model must be built to exploit it. This agrees with the simulation, where PointACT and Fast-WAM reach 33.7% and 52.9% against 69.0% for PointWAM (Table 1), and where removing scene forecasting with the input fixed costs 10.9 pt (Table 4 (b)). We added these results to Fig. 5 and Appendix D.7 of the revision.

AuthorsOpenReview ↗

Response to reviewer uBvz [3/3]

(3) Attributing the gain from pre-training (W3)

To separate the gain from the initialization, we removed the robot pre-training from GR00T N1.6 and repeated the experiment of Fig. 6, fine-tuning each model on the same DexJoCo demonstrations with and without EgoDex pre-training.

  • GR00T N1.6 w/o pre-trained action head trains the action head from scratch. The vision-language model keeps its pre-trained weights, including 4 language-model layers tuned on robot data.
  • GR00T N1.6 w/o pre-trained action head and VLM also trains the language model and its projector from scratch. Only its vision encoder and text embeddings stay pre-trained, as only the scene and text encoders do in PointWAM.
ModelSR w/o pre-training (%) ↑SR with pre-training (%) ↑Gain (pt) ↑
GR00T N1.657.3 ± 1.856.6 ± 2.0−0.7
GR00T N1.6 w/o pre-trained action head53.2 ± 1.755.5 ± 1.9+2.3
GR00T N1.6 w/o pre-trained action head and VLM47.0 ± 1.850.6 ± 1.8+3.6
PointWAM12.1 ± 1.160.1 ± 3.1+48.0

Mean ± standard deviation over three evaluation seeds.

The gap remains without the robot pre-training. GR00T N1.6 then drops from 57.3% to 47.0%, which leaves it room to improve, yet EgoDex pre-training raises it by only 3.6 pt, whereas PointWAM gains 48.0 pt. The gain therefore does not follow from a lower starting point alone. It is consistent with the shared 3D representation, in which human and robot demonstrations supervise the same scene and hand trajectories. We added this analysis to Appendix D.6 of the revision.

(4) Real-world evaluation on more tasks and platforms (W4)

We added a second platform, RB-Y1 with two five-finger 20-DoF Wuji hands, which differs from the first in both the arm and the hand. Its task, paper-roll replacement, needs both hands in sequence, as one hand pulls the empty core off a holder and the other mounts a new roll (24 trials per method and task).

PlatformTaskπ0.5 SR (%) ↑GR00T N1.6 SR (%) ↑PointWAM SR (%) ↑
OpenArm + Inspire RH56F1Banana pick-and-place58.350.070.8
OpenArm + Inspire RH56F1Square-peg insertion25.016.750.0
RB-Y1 + Wuji (new)Paper-roll replacement25.020.841.7

Banana pick-and-place is reported over all 24 trials, in-domain and out-of-domain together.

PointWAM leads on all three tasks, by 12.5 to 25.0 pt over the stronger baseline. In simulation, we also added 13 bimanual Bench2Dex tasks on seven dexterous hands, where PointWAM reaches 55.2% against 35.5% for the strongest baseline (Table 2). We added the new platform and task to Sec. 4.1 and 4.2 of the revision.

Reviewer_Z1LDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces PointWAM, a method for learning robotic dexterous manipulation from human demonstration videos. The method jointly predicts the 3D trajectories of scene points and hand keypoints within a shared space–time coordinate frame and generates robot actions through an action retargeter. Training consists of pretraining on EgoDex and VITRA, followed by fine-tuning on robot demonstrations. Scene trajectories provide auxiliary supervision, whereas action prediction directly uses hand trajectories and robot states. Experiments on DexJoCo support the benefits of pretraining on human demonstrations, while real-robot experiments demonstrate the method’s manipulation performance.

Critical strengths
  1. The shared geometric representation provides a unified interface for scene and action modeling. Representing scene points and semantically corresponding hand keypoints in a common 3D prediction space provides a consistent interface for learning from human and robot demonstrations.

  2. The performance gains on DexJoCo receive substantial experimental support. The paper also provides a relatively comprehensive set of ablations, including analyses of voxel-related design choices.

  3. The core method is clearly presented and easy to follow.

Critical weaknesses
  1. The quality of the world predictions lacks quantitative evaluation. The paper reports task-success ablations for scene-trajectory supervision and provides qualitative trajectory visualizations, but it does not report prediction errors for scene or hand trajectories. It also does not quantify the consistency of contact relationships in the predicted trajectories.

  2. The applicability of the hand-keypoint mapping across different hand morphologies remains unclear. A key design choice is to establish correspondences between human-hand and robot-hand keypoints. However, it remains unclear whether this mapping remains appropriate when the robotic hand differs substantially from the human hand in size or kinematic structure.

Questions for authors
  1. Do the authors evaluate transfer across dexterous-hand platforms with different morphologies and sizes? In particular, how does transfer performance compare between human-sized hands and hands with substantially larger size?
3 comments
AuthorsOpenReview ↗

Response to reviewer Z1LD

Dear reviewer Z1LD, we thank you for recognizing our shared geometric representation as a unified interface for human and robot demonstrations, and our comprehensive ablations. Please refer to the general response for a summary of the revision. Here, we address your specific concerns.

(1) Quantitative evaluation of the world predictions (W1)

We held out 10% of the DexJoCo demonstrations (97 of 959) and measured two metrics.

  • Prediction error: the ℓ2 error of moving points (displacement > 1 cm), following PointWorld (Huang et al., 2026), averaged over the 0.6 s horizon and at its end.
  • Contact consistency: contact F1, whether each fingertip is within 2 cm of the moving object in the same frames as in the ground truth.
ForecastHand error: average / final (cm) ↓Scene error: average / final (cm) ↓Contact F1 (0–1) ↑
No forecast–4.16 / 7.730.39
PointWAM w/o pre-training2.74 / 4.712.49 / 4.350.71
PointWAM2.63 / 4.532.36 / 4.110.76

"No forecast" keeps every scene point at its initial position, and its contact F1 is computed with the ground-truth hand trajectories.

PointWAM predicts the scene with 43% lower error than no forecast, while static scene points stay within 0.1 cm. Its contact F1 is 0.76 against 0.39, which shows that the predicted object moves with the predicted hand. Pre-training on human videos improves every metric. We added these results to Sec. 4.4 (Table 5) and Appendix D.3 of the revision.

(2) Transfer across hand morphologies and sizes (W2 & Q1)

The paper covered two hands, the four-finger Allegro hand on DexJoCo and the five-finger Inspire RH56F1 hand on the real robot. We added seven hands from Bench2Dex (Yang et al., 2026b) and the five-finger Wuji hand on a second real robot. The table lists the real-robot hands and then the simulated hands by hand length, with the success rate of PointWAM without and with pre-training on human videos.

HandFingersHand DoFHand length (cm)BenchmarkSR w/o pre-training (%) ↑SR with pre-training (%) ↑Gain (pt) ↑
Human hand52118.5Pre-training–––
Inspire RH56F15617.6Real robot–60.4–
Wuji52019.5Real robot (new)–41.7–
ORCA51318.7Bench2Dex (new)1.544.5+43.0
Sharpa Wave52020.4Bench2Dex (new)1.511.0+9.5
Schunk SVH5921.5Bench2Dex (new)46.586.0+39.5
Inspire RH5DG251321.6Bench2Dex (new)30.584.5+54.0
DexHand02152022.4Bench2Dex (new)18.535.0+16.5
LEAP41323.2Bench2Dex (new)17.077.0+60.0
Allegro41624.4DexJoCo12.169.0+56.9

Hand length runs from the base of the palm to the middle fingertip (human hand: median over EgoDex). The human hand DoF excludes the 6 DoF of the wrist (Lin et al., 2000). The six Bench2Dex hands are those that also have a run without pre-training, each on one task. DexJoCo covers ten tasks, the Inspire RH56F1 hand two, and the Wuji hand one.

Pre-training raises the success rate on every hand we measured, and the gain does not shrink with hand size. The two largest hands, LEAP and Allegro, are 1.3× longer than the human hand and have four fingers, yet they gain 60.0 and 56.9 pt, as much as the human-sized ORCA hand (43.0 pt). The mapping matches only the palm and the fingertips, so a hand need not share size, joints, or finger count with the human hand. Over all 13 Bench2Dex tasks on seven hands, PointWAM reaches 55.2%, above every baseline of the benchmark, the strongest being GR00T N1.5 at 35.5% (Table 2). We added these results to Sec. 4.2 (Table 2), Sec. 4.3, and Appendix D.4 (Table 10) of the revision.

Thank you for the detailed response and additional experimental results.

The quantitative evaluation provides a clearer assessment of the world predictions, while the experiments across different dexterous-hand morphologies offer further evidence of the method's effectiveness. These results largely address my concerns.

I am inclined to maintain my Weak Accept rating and would encourage the authors to incorporate the additional results into the final version of the paper.

Thank you for reading our response and for your constructive feedback. We are glad that the quantitative evaluation of the world predictions and the experiments across dexterous-hand morphologies addressed your concerns. We will incorporate these results into the final version of the paper, and we are happy to clarify any remaining questions during the discussion period.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank all reviewers for their valuable comments and constructive feedback. We are grateful that the reviewers recognized the following strengths of our work: (1) the unified 3D representation for human and robot demonstrations, (2) the strong results, and (3) the comprehensive experiments.

We revised the manuscript according to the feedback and marked every change in blue. The main updates are summarized below, and we respond to each reviewer's comments in the corresponding thread.

(1) New evaluations

  • Bench2Dex: 13 bimanual tasks on seven dexterous hands (Sec. 4.1, 4.2, Table 2, App. D.8).
  • Real robot: A second platform, RB-Y1 + Wuji, and two more baselines, PointACT and Fast-WAM (Sec. 4.1, 4.2, Fig. 5, App. D.7).

(2) New analyses

  • Trajectory forecasting: Quantitative evaluation on held-out demonstrations (Sec. 4.4, Table 5, App. D.3).
  • Transfer across hands: Human pre-training across nine hands of different morphologies and sizes (Sec. 4.3, App. D.4, Table 10).
  • Pre-training: Per-task gains, failure analysis on RoboDojo-Precision, and the gain without robot pre-trained weights (App. D.5, D.6).
  • Action execution: The execution setting on the real robot (App. C.4).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.