- Rating
- 3(+2)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
DexOmni introduces a large-scale simulation dataset and benchmark for dexterous robotic manipulation. It provides 18,000 human-teleoperated demonstrations (~56 hours) collected in Isaac Sim, covering 50 training tasks involving functional tool use, articulated objects, precision manipulation, and bimanual coordination. The authors evaluate nine policy approaches, including Diffusion Policy, five vision-language-action models, and three world-action models, using task success rates and fine-grained task-stage diagnostics. Experiments show that generalization to held-out tasks remains extremely limited. Overall, the paper contributes a substantial human-collected simulation dataset, a diverse dexterous manipulation benchmark, and systematic evaluation of current policy limitations, rather than proposing a new policy learning algorithm.
Critical strengths
-
Substantial human teleoperation effort. Getting 18,000 curated demonstrations is expensive even in simulation, especially for dexterous manipulation requiring precise contacts and bimanual coordination. This is a meaningful dataset contribution.
-
Strong policy training baselines. The paper evaluates nine approaches spanning VLAs and WAMs. This provides relatively broad coverage of important contemporary policy formulations.
-
Thoughtful evaluation protocols. Goes beyond binary success rates with task-stage progress, contact-related failure analysis, temporal coordination, goal maintenance, and standardized evaluation budgets.
Critical weaknesses
-
Limited task diversity and weak generalization. Despite 50 training tasks, performance on unseen tasks is poor. The best model (OpenWAM) achieves only 0.9% success on asset/relation transfer and 0% on skill composition. This suggests that the demonstrated skills do not transfer effectively to new tasks, which is a direct indicator that the dataset diversity is still lacking.
-
Rigid-body simulation limits physical fidelity. The Wuji Hand has soft surfaces, but the paper does not establish whether fingertip deformation is modeled. It appears to be using rigid contact approximations, which means resulting contact dynamics would differ substantially from reality. This potentially limits both the usefulness of simulated training data and the credibility of simulation-only evaluation, especially for contact-rich dexterous skills. The extent of this issue remains unverified.
-
Important simulation implementation details are missing. The paper specifies Isaac Sim 5.1 but does not adequately describe the robot-hand contact model, compliant materials, physics parameters, or deformable-object simulation methods. These omissions make it difficult to assess physical realism and reproducibility.
-
Limited evidence that simulation data improves real-world learning. Quantitative policy evaluations are predominantly simulation-based. The real-world Place Ball in Bowl experiment reports qualitative improvement from sim+real co-training but lacks controlled success-rate comparisons, training configurations, and model initialization details. In particular, fine-tuning a pretrained VLA on real teleoperation data alone might achieve comparable or better results, making the incremental value of simulation demonstrations unclear.
Questions for authors
-
Robot-hand physics: Is the Wuji Hand simulated entirely with rigid articulated links, or are compliant fingertips/soft pads explicitly modeled? How are contact stiffness and friction calibrated against hardware?
-
Deformable simulation: Which PhysX deformable solvers and material models are used for the sponge, towel, and pillow? W
-
Sim-to-real transfer: What are the quantitative real-world success rates across representative tasks, particularly contact-rich manipulation?
-
Sim+real co-training: Are the policies trained from scratch or initialized from pretrained foundation models? What architecture, data ratios, and training budgets are used? How does co-training compare against fine-tuning the same pretrained VLA exclusively on real demonstrations?
-
Generalization: Why is held-out task performance nearly zero? Is this primarily due to insufficient task coverage, embodiment-specific simulation data, or limitations of the evaluated policies?
11 comments
Response to Reviewer QKkY (2/7)
W1 (continued): Data quality
The data are of high quality. All 18,000 demonstrations are curated human teleoperation recorded in the evaluation simulator and re-rendered with physically based materials and lighting, and policies trained with them execute on the physical platform (supplementary real-robot videos). Compared with established simulation benchmarks, DexOmni covers a broad range of dexterous task types with the largest human corpus:
| Benchmark | Embodiment | Task types | Tasks | Trajectories | Separate held-out test set |
|---|---|---|---|---|---|
| LIBERO [1] | one-arm gripper | spatial, object, goal and long-horizon household tasks | 130 | 6,500 human | No |
| CALVIN [2] | one-arm gripper | language-conditioned long-horizon tabletop tasks | 34 | 24 h of human play | Yes, same tasks (unseen environment and instructions) |
| RoboTwin 2.0 [3] | two-arm gripper | bimanual manipulation | 50 | over 100,000 generated | Yes, same tasks (unseen scene randomization and instructions) |
| RoboCasa [4] | mobile one-arm gripper | everyday kitchen tasks, atomic and composite | 100 | 1,250 human, 100,000 generated | Yes, same tasks (unseen objects and kitchen styles) |
| ManiSkill2 [5] | gripper | rigid- and soft-body manipulation skills | 20 task families | over 30,000 generated | Yes, same tasks (unseen object sets) |
| DexJoCo [6] | dexterous hands | task-oriented dexterous manipulation | 11 | 1,100 human | Perturbed trained tasks only (visual, language) |
| DexVerse [7] | six dexterous hands | multi-task dexterous manipulation | 100 | 3,180 human | Perturbed trained tasks only (visual, embodiment) |
| Bench2Dex [8] | twelve dexterous hands | visuo-tactile bimanual dexterous manipulation | 26 | about 1,300 human | Perturbed trained tasks only (visual, geometric) |
| DexOmni | two arms with 20-DoF hands | functional tool use, numerical and spatial grounding, articulation, bimanual coordination, deformable objects | 50 + 16 | 18,000 human (about 56 h) | Yes, new tasks: 16 test-only tasks with no demonstrations, plus held-out material, lighting and wording |
Of these benchmarks, DexOmni has the most human demonstrations and is the only one reserving whole tasks for testing. Table 15 also lists RoboDojo (8 held-out tasks); comparisons appear in Appendix B.13 and Table 1.
Policies trained on these data lose little under held-out appearance and wording. We evaluated all eight checkpoints on paired draws (50 per task, both arms run fresh in the same window), changing one factor at a time to a setting never seen in training:
| Held-out condition | In-distribution | Held-out | Pooled change, points (95% CI) | Range over checkpoints |
|---|---|---|---|---|
| Table material outside the training bank (11 tasks) | 33.5% | 30.2% | −3.2 (−4.7, −1.7) | −6.7 to −0.4 |
| Lighting beyond the training ranges (11 tasks) | 33.5% | 32.8% | −0.6 (−2.0, +0.7) | −4.7 to +5.3 |
| Main-camera pose 5× outside the training jitter (11 tasks) | 33.5% | 31.9% | −1.6 (−3.0, −0.2) | −6.0 to +8.5 |
| Held-out paraphrases (4 language tasks) | 41.7% | 38.9% | −2.8 (−4.8, −0.7) | −5.5 to +2.5 |
Success is task-macro for the visual shifts (550 paired episodes per checkpoint and condition) and pooled over 1,599 pairs for paraphrases; intervals are task-stratified paired bootstraps. Under the visual shifts the ranking holds (26, 27 and 27 of 28 pairwise orders preserved), and episode-level agreement between any two wordings (72–90%) matches that between two canonical runs of the same draws (73–90%). We have included these results in Appendix B.10 of revision.
[1] B. Liu, Y. Zhu, C. Gao, et al. LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning. arXiv:2306.03310, 2023.
[2] O. Mees, L. Hermann, E. Rosete-Beas, W. Burgard. CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks. arXiv:2112.03227, 2022.
[3] T. Chen, Z. Chen, B. Chen, et al. RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulation. arXiv:2506.18088, 2025.
[4] S. Nasiriany, A. Maddukuri, L. Zhang, et al. RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots. RSS 2024.
[5] J. Gu, F. Xiang, X. Li, et al. ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills. arXiv:2302.04659, 2023.
[6] H. Wang, W. Zhao, X. Wang, et al. DexJoCo: A Benchmark and Toolkit for Task-Oriented Dexterous Manipulation on MuJoCo. arXiv:2605.16257, 2026.
[7] Y. Yao, Z. Xu, T. Zhang, et al. DexVerse: A Modular Benchmark for Multi-Task, Multi-Embodiment Dexterous Manipulation. arXiv:2607.08751, 2026.
[8] Z. Yang, Y. Zhang, D. Zhang, et al. Bench2Dex: Benchmarking Visuo-Tactile Bimanual Dexterous Manipulation Across Dexterous Hands. arXiv:2609.15726, 2026.
Response to Reviewer QKkY (3/7)
W1 (continued): Known generalization limits of VLAs and WAMs
High-level generalization is a widely reported weakness of current VLAs and WAMs. Independent studies find that policies which succeed on trained tasks drop sharply, often to near zero, on new tasks and on new combinations of skills they have already been shown:
| Study | Policies | Trained tasks or combinations | New tasks or combinations |
|---|---|---|---|
| CRAFT [1] | π0.5 [2] / π0 [3] | 100.0 / 99.7% | 0.7 / 0.3% on undemonstrated combinations of demonstrated skills |
| ACG-Bench [4] | π0.5 on two arms | 37.0% in-domain | 2.9% on 17 unseen arm-wise compositions |
| LIBERO-PRO [5] | VLAs on LIBERO | over 90% | 0.0% under perturbed objects, initial states, instructions and environments |
| AGNOSTOS [6] | π0 / OpenVLA [7] | trained on diverse datasets | 17.5 / 13.6% on 23 unseen tasks |
| AdaWAM [8] | FastWAM [9] | trained LIBERO tasks | 0% on an unseen subtask combination |
RT-2 [10] already observed that web-scale pretraining gives the robot no new motions, so its physical skills stay within those seen in the robot data. AGNOSTOS finds that current VLAs struggle on unseen tasks despite being trained on diverse datasets, and LIBERO-PRO traces the collapse to rote memorization of action sequences and layouts. For WAMs, cross-task generalization likewise remains difficult because predicted futures collapse on unseen tasks [11], and out of distribution, video generation models imitate the closest training case rather than the underlying rule [12]. π0.5 and FastWAM, both evaluated in DexOmni, are among the policies that fail in these studies. We have included this discussion in Appendix D.1 of revision.
Near-zero Transfer-16 success thus matches a documented limitation of these policy classes: it measures their limited ability to carry learned skills to new assets, relations and sequences, not a lack of data quality or diversity.
[1] T. Yang, Y. Na, Y. Cho, et al. Same Scene, Different Task: Skill Alignment for Compositional Generalization in VLAs. arXiv:2610.00524, 2026.
[2] Physical Intelligence, K. Black, N. Brown, et al. π0.5: a Vision-Language-Action Model with Open-World Generalization. arXiv:2504.16054, 2025.
[3] K. Black, N. Brown, D. Driess, et al. π0: A Vision-Language-Action Flow Model for General Robot Control. arXiv:2410.24164, 2024.
[4] Z. Zhang, B. Ran, Y. Wu, et al. Arm-wise Compositional Generalization in Dual-Arm Vision-Language-Action Models. arXiv:2610.06184, 2026.
[5] X. Zhou, Y. Xu, G. Tie, et al. LIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models Beyond Memorization. arXiv:2510.03827, 2025.
[6] J. Zhou, K. Ye, J. Liu, et al. Exploring the Limits of Vision-Language-Action Manipulations in Cross-task Generalization. arXiv:2505.15660, 2025.
[7] M. J. Kim, K. Pertsch, S. Karamcheti, et al. OpenVLA: An Open-Source Vision-Language-Action Model. CoRL 2024.
[8] Y. Tang, J. Xu, Y. Shang, et al. Dreaming when Necessary: Advancing World Action Models with Adaptive Multi-Modal Reasoning. arXiv:2606.07089, 2026.
[9] T. Yuan, Z. Dong, Y. Liu, H. Zhao. Fast-WAM: Do World Action Models Need Test-time Future Imagination? arXiv:2603.16666, 2026.
[10] A. Brohan, N. Brown, J. Carbajal, et al. RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control. CoRL 2023.
[11] C. Zhao, Z. Ye, Y. Lin, et al. What Limits Cross-Task Generalization in World Action Models? arXiv:2610.16641, 2026.
[12] B. Kang, Y. Yue, R. Lu, et al. How Far is Video Generation from World Model: A Physical Law Perspective. ICML 2025.
Response to Reviewer QKkY (1/7)
Thank you for your detailed review. We appreciate your recognition of the substantial teleoperation effort behind the 18,000 curated demonstrations, the broad coverage of contemporary policies and the evaluation protocols that go beyond binary success. We take your concerns seriously, especially about generalization to held-out tasks and the physical fidelity of the simulation. In the point-by-point response below, we directly address each of these issues, and we hope these clarifications alleviate the concerns underlying your current assessment.
W1: Task diversity and near-zero held-out success
Near-zero Transfer-16 success points to the current policies, not insufficient dataset diversity, for four reasons: (1) Transfer-16 is evaluation-only by design, and the skills its asset, relation and composition tasks require are present in the data and are learned; (2) a strong model that never sees DexOmni data is also near zero on Transfer-16, while every DexOmni-trained model exceeds it on Core-50; (3) the data are of high quality: they exceed established benchmarks in human demonstrations, and policies trained on them lose little under held-out appearance and wording; (4) poor high-level generalization is a widely documented limitation of current VLAs and WAMs. As Reviewer 9H8w also notes, human verification rules out simple task infeasibility.
The skills are in the data and are learned. Every one of the 66 tasks is verified in person by the data-collection operators, and the demonstrations pass a written review with separated roles: a new task enters collection only after a hand-reviewed pilot of at least 20 episodes with at least 98% agreement between labels and objective completion evidence, every episode is checked automatically, and blind reviewers cover about 20% of each batch and re-review a further 5–10% (Appendix B.2). For π0.5 [1] and OpenWAM [2], the same native checker transition succeeds less often in the target than in the source in 28 of 33 comparable cases: a left-hand pickup succeeds in 72–88% of their Pass Block episodes but in 0–2% of Pass Glue Stick episodes, and pouring the first seasoning succeeds in 83–97% of their Pour Seasonings episodes but in 0–8% of Season and Stir episodes.
Low transfer success is not specific to DexOmni data. We evaluated two frontier models, GPT-6 and Fable 5.2, zero-shot on all 66 tasks. Neither receives DexOmni demonstrations or finetuning; each observes three RGB views, the measured robot state and the instruction, and controls both hands through a single move_eef function that directly sets absolute palm poses and 20 finger-joint targets per hand, executed by the same native controllers, time limits and checkers, with 10 trials per task scored within 100 model responses per trial.
The equal totals come from different successes: both models run the same 660 fixed trials, yet only 6 of the 29 successes of each fall on the same trial, close to the 6.6 expected if they succeeded independently on each task; their success counts differ on all 19 tasks that either model solves, and GPT-6 leads on contact-rich tasks (7.3% versus 3.3%) while Fable 5.2 leads on primitive ones (11.2% versus 7.6%). Zero-shot, GPT-6 and Fable 5.2 solve 12 and 14 of the 66 tasks, including Transfer-16 tasks, which shows strong task understanding; yet both reach only 1.9% on Transfer-16 and succeed in only 2 of 180 Core-50 functional-task trials (1.1%), because emitting high-dimensional palm-and-finger coordinates directly is still hard for them. Near-zero transfer is therefore not specific to policies trained on DexOmni data. Every model trained on DexOmni demonstrations exceeds both zero-shot models on Core-50, from 10.1% for single-task DP [4] to 29.1% for OpenWAM, which shows the value of the data.
We have included the GPT-6 and Fable 5.2 references and the stage-level analysis in Section 5.3 and Appendix D of revision.
[1] Physical Intelligence, K. Black, N. Brown, et al. π0.5: a Vision-Language-Action Model with Open-World Generalization. arXiv:2504.16054, 2025.
[2] Y. Wang, S. Huang, M. Li, et al. OpenWAM: An Open, Modular Exploration Towards Systematic World-Action Model Pretraining. arXiv:2609.07398, 2026.
[3] E. Yu, H. Lv, J. Sun, et al. DM0: An Embodied-Native Vision-Language-Action Model towards Physical AI. arXiv:2602.14974, 2026.
[4] C. Chi, Z. Xu, S. Feng, et al. Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. RSS 2023.