acceptodds
Under review as a conference paper at ICLR 2027

DexOmni: A Large-Scale Simulation Benchmark for Generalizable Dexterous Manipulation

Abstract

Dexterous manipulation requires reliable contact establishment, temporal coordination of multi-finger and bimanual actions, as well as maintenance of achieved goals, while also transferring these abilities to unfamiliar objects, interactive relationships, and skill combinations. Herein, we introduce DexOmni, a simulation data platform and benchmark featuring 66 tasks whose feasibility was verified through human demonstrations, 18,000 curated human demonstrations, high-fidelity re-rendering, and visual and goal-preserving prompt randomization. Covering a diverse range of interaction types, task structures, and asset types, core-50 provides demonstrations and execution tests for 50 tasks spanning functional tool use, numerical and spatial grounding, articulation, and bimanual manipulation. Transfer-16 reserves 16 tasks to evaluate asset and relation transfer, skill composition, and deformable manipulation without target-task demonstrations in the benchmark training set. We compare task-specialized diffusion policies, vision-language-action models, and world-action models using success and native-stage diagnostics. Across 40,000 Core-50 episodes, the best of eight multi-task checkpoints reaches 29.1% task-macro success. Success rates are 35–51% for pushing, articulation, and pick-and-place but only 2–3% for tool use and precision fitting; this gap persists after controlling for demonstration count and duration. Failures in contact establishment, temporal coordination, and goal maintenance emerge as the main bottlenecks in multi-finger and bimanual control. Transfer-16 leaves the most headroom; ordered progress separates its checkpoints, and compositions add another failure: policies repeat finished steps or start later ones early. Overall, DexOmni provides a challenging benchmark for systematically evaluating the reliability and generalization of embodied policies for dexterous robotic hands.

Primary area:
datasets and benchmarks
Keywords:
benchmark, dexterous hand, simulation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_QKkYOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

DexOmni introduces a large-scale simulation dataset and benchmark for dexterous robotic manipulation. It provides 18,000 human-teleoperated demonstrations (~56 hours) collected in Isaac Sim, covering 50 training tasks involving functional tool use, articulated objects, precision manipulation, and bimanual coordination. The authors evaluate nine policy approaches, including Diffusion Policy, five vision-language-action models, and three world-action models, using task success rates and fine-grained task-stage diagnostics. Experiments show that generalization to held-out tasks remains extremely limited. Overall, the paper contributes a substantial human-collected simulation dataset, a diverse dexterous manipulation benchmark, and systematic evaluation of current policy limitations, rather than proposing a new policy learning algorithm.

Critical strengths
  1. Substantial human teleoperation effort. Getting 18,000 curated demonstrations is expensive even in simulation, especially for dexterous manipulation requiring precise contacts and bimanual coordination. This is a meaningful dataset contribution.

  2. Strong policy training baselines. The paper evaluates nine approaches spanning VLAs and WAMs. This provides relatively broad coverage of important contemporary policy formulations.

  3. Thoughtful evaluation protocols. Goes beyond binary success rates with task-stage progress, contact-related failure analysis, temporal coordination, goal maintenance, and standardized evaluation budgets.

Critical weaknesses
  1. Limited task diversity and weak generalization. Despite 50 training tasks, performance on unseen tasks is poor. The best model (OpenWAM) achieves only 0.9% success on asset/relation transfer and 0% on skill composition. This suggests that the demonstrated skills do not transfer effectively to new tasks, which is a direct indicator that the dataset diversity is still lacking.

  2. Rigid-body simulation limits physical fidelity. The Wuji Hand has soft surfaces, but the paper does not establish whether fingertip deformation is modeled. It appears to be using rigid contact approximations, which means resulting contact dynamics would differ substantially from reality. This potentially limits both the usefulness of simulated training data and the credibility of simulation-only evaluation, especially for contact-rich dexterous skills. The extent of this issue remains unverified.

  3. Important simulation implementation details are missing. The paper specifies Isaac Sim 5.1 but does not adequately describe the robot-hand contact model, compliant materials, physics parameters, or deformable-object simulation methods. These omissions make it difficult to assess physical realism and reproducibility.

  4. Limited evidence that simulation data improves real-world learning. Quantitative policy evaluations are predominantly simulation-based. The real-world Place Ball in Bowl experiment reports qualitative improvement from sim+real co-training but lacks controlled success-rate comparisons, training configurations, and model initialization details. In particular, fine-tuning a pretrained VLA on real teleoperation data alone might achieve comparable or better results, making the incremental value of simulation demonstrations unclear.

Questions for authors
  1. Robot-hand physics: Is the Wuji Hand simulated entirely with rigid articulated links, or are compliant fingertips/soft pads explicitly modeled? How are contact stiffness and friction calibrated against hardware?

  2. Deformable simulation: Which PhysX deformable solvers and material models are used for the sponge, towel, and pillow? W

  3. Sim-to-real transfer: What are the quantitative real-world success rates across representative tasks, particularly contact-rich manipulation?

  4. Sim+real co-training: Are the policies trained from scratch or initialized from pretrained foundation models? What architecture, data ratios, and training budgets are used? How does co-training compare against fine-tuning the same pretrained VLA exclusively on real demonstrations?

  5. Generalization: Why is held-out task performance nearly zero? Is this primarily due to insufficient task coverage, embodiment-specific simulation data, or limitations of the evaluated policies?

11 comments
AuthorsOpenReview ↗

Response to Reviewer QKkY (1/7)

Thank you for your detailed review. We appreciate your recognition of the substantial teleoperation effort behind the 18,000 curated demonstrations, the broad coverage of contemporary policies and the evaluation protocols that go beyond binary success. We take your concerns seriously, especially about generalization to held-out tasks and the physical fidelity of the simulation. In the point-by-point response below, we directly address each of these issues, and we hope these clarifications alleviate the concerns underlying your current assessment.

W1: Task diversity and near-zero held-out success

Near-zero Transfer-16 success points to the current policies, not insufficient dataset diversity, for four reasons: (1) Transfer-16 is evaluation-only by design, and the skills its asset, relation and composition tasks require are present in the data and are learned; (2) a strong model that never sees DexOmni data is also near zero on Transfer-16, while every DexOmni-trained model exceeds it on Core-50; (3) the data are of high quality: they exceed established benchmarks in human demonstrations, and policies trained on them lose little under held-out appearance and wording; (4) poor high-level generalization is a widely documented limitation of current VLAs and WAMs. As Reviewer 9H8w also notes, human verification rules out simple task infeasibility.

The skills are in the data and are learned. Every one of the 66 tasks is verified in person by the data-collection operators, and the demonstrations pass a written review with separated roles: a new task enters collection only after a hand-reviewed pilot of at least 20 episodes with at least 98% agreement between labels and objective completion evidence, every episode is checked automatically, and blind reviewers cover about 20% of each batch and re-review a further 5–10% (Appendix B.2). For π0.5 [1] and OpenWAM [2], the same native checker transition succeeds less often in the target than in the source in 28 of 33 comparable cases: a left-hand pickup succeeds in 72–88% of their Pass Block episodes but in 0–2% of Pass Glue Stick episodes, and pouring the first seasoning succeeds in 83–97% of their Pour Seasonings episodes but in 0–8% of Season and Stir episodes.

Low transfer success is not specific to DexOmni data. We evaluated two frontier models, GPT-6 and Fable 5.2, zero-shot on all 66 tasks. Neither receives DexOmni demonstrations or finetuning; each observes three RGB views, the measured robot state and the instruction, and controls both hands through a single move_eef function that directly sets absolute palm poses and 20 finger-joint targets per hand, executed by the same native controllers, time limits and checkers, with 10 trials per task scored within 100 model responses per trial.

SuiteGPT-6 zero-shotFable 5.2 zero-shotBest DexOmni-trained policy
Core-505.2% (26/500)5.2% (26/500)29.1% (OpenWAM [2])
Transfer-161.9% (3/160)1.9% (3/160)0.9% (DM0.5 [3])

The equal totals come from different successes: both models run the same 660 fixed trials, yet only 6 of the 29 successes of each fall on the same trial, close to the 6.6 expected if they succeeded independently on each task; their success counts differ on all 19 tasks that either model solves, and GPT-6 leads on contact-rich tasks (7.3% versus 3.3%) while Fable 5.2 leads on primitive ones (11.2% versus 7.6%). Zero-shot, GPT-6 and Fable 5.2 solve 12 and 14 of the 66 tasks, including Transfer-16 tasks, which shows strong task understanding; yet both reach only 1.9% on Transfer-16 and succeed in only 2 of 180 Core-50 functional-task trials (1.1%), because emitting high-dimensional palm-and-finger coordinates directly is still hard for them. Near-zero transfer is therefore not specific to policies trained on DexOmni data. Every model trained on DexOmni demonstrations exceeds both zero-shot models on Core-50, from 10.1% for single-task DP [4] to 29.1% for OpenWAM, which shows the value of the data.

We have included the GPT-6 and Fable 5.2 references and the stage-level analysis in Section 5.3 and Appendix D of revision.

[1] Physical Intelligence, K. Black, N. Brown, et al. π0.5: a Vision-Language-Action Model with Open-World Generalization. arXiv:2504.16054, 2025.

[2] Y. Wang, S. Huang, M. Li, et al. OpenWAM: An Open, Modular Exploration Towards Systematic World-Action Model Pretraining. arXiv:2609.07398, 2026.

[3] E. Yu, H. Lv, J. Sun, et al. DM0: An Embodied-Native Vision-Language-Action Model towards Physical AI. arXiv:2602.14974, 2026.

[4] C. Chi, Z. Xu, S. Feng, et al. Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. RSS 2023.

AuthorsOpenReview ↗

Response to Reviewer QKkY (2/7)

W1 (continued): Data quality

The data are of high quality. All 18,000 demonstrations are curated human teleoperation recorded in the evaluation simulator and re-rendered with physically based materials and lighting, and policies trained with them execute on the physical platform (supplementary real-robot videos). Compared with established simulation benchmarks, DexOmni covers a broad range of dexterous task types with the largest human corpus:

BenchmarkEmbodimentTask typesTasksTrajectoriesSeparate held-out test set
LIBERO [1]one-arm gripperspatial, object, goal and long-horizon household tasks1306,500 humanNo
CALVIN [2]one-arm gripperlanguage-conditioned long-horizon tabletop tasks3424 h of human playYes, same tasks (unseen environment and instructions)
RoboTwin 2.0 [3]two-arm gripperbimanual manipulation50over 100,000 generatedYes, same tasks (unseen scene randomization and instructions)
RoboCasa [4]mobile one-arm grippereveryday kitchen tasks, atomic and composite1001,250 human, 100,000 generatedYes, same tasks (unseen objects and kitchen styles)
ManiSkill2 [5]gripperrigid- and soft-body manipulation skills20 task familiesover 30,000 generatedYes, same tasks (unseen object sets)
DexJoCo [6]dexterous handstask-oriented dexterous manipulation111,100 humanPerturbed trained tasks only (visual, language)
DexVerse [7]six dexterous handsmulti-task dexterous manipulation1003,180 humanPerturbed trained tasks only (visual, embodiment)
Bench2Dex [8]twelve dexterous handsvisuo-tactile bimanual dexterous manipulation26about 1,300 humanPerturbed trained tasks only (visual, geometric)
DexOmnitwo arms with 20-DoF handsfunctional tool use, numerical and spatial grounding, articulation, bimanual coordination, deformable objects50 + 1618,000 human (about 56 h)Yes, new tasks: 16 test-only tasks with no demonstrations, plus held-out material, lighting and wording

Of these benchmarks, DexOmni has the most human demonstrations and is the only one reserving whole tasks for testing. Table 15 also lists RoboDojo (8 held-out tasks); comparisons appear in Appendix B.13 and Table 1.

Policies trained on these data lose little under held-out appearance and wording. We evaluated all eight checkpoints on paired draws (50 per task, both arms run fresh in the same window), changing one factor at a time to a setting never seen in training:

Held-out conditionIn-distributionHeld-outPooled change, points (95% CI)Range over checkpoints
Table material outside the training bank (11 tasks)33.5%30.2%−3.2 (−4.7, −1.7)−6.7 to −0.4
Lighting beyond the training ranges (11 tasks)33.5%32.8%−0.6 (−2.0, +0.7)−4.7 to +5.3
Main-camera pose 5× outside the training jitter (11 tasks)33.5%31.9%−1.6 (−3.0, −0.2)−6.0 to +8.5
Held-out paraphrases (4 language tasks)41.7%38.9%−2.8 (−4.8, −0.7)−5.5 to +2.5

Success is task-macro for the visual shifts (550 paired episodes per checkpoint and condition) and pooled over 1,599 pairs for paraphrases; intervals are task-stratified paired bootstraps. Under the visual shifts the ranking holds (26, 27 and 27 of 28 pairwise orders preserved), and episode-level agreement between any two wordings (72–90%) matches that between two canonical runs of the same draws (73–90%). We have included these results in Appendix B.10 of revision.

[1] B. Liu, Y. Zhu, C. Gao, et al. LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning. arXiv:2306.03310, 2023.

[2] O. Mees, L. Hermann, E. Rosete-Beas, W. Burgard. CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks. arXiv:2112.03227, 2022.

[3] T. Chen, Z. Chen, B. Chen, et al. RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulation. arXiv:2506.18088, 2025.

[4] S. Nasiriany, A. Maddukuri, L. Zhang, et al. RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots. RSS 2024.

[5] J. Gu, F. Xiang, X. Li, et al. ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills. arXiv:2302.04659, 2023.

[6] H. Wang, W. Zhao, X. Wang, et al. DexJoCo: A Benchmark and Toolkit for Task-Oriented Dexterous Manipulation on MuJoCo. arXiv:2605.16257, 2026.

[7] Y. Yao, Z. Xu, T. Zhang, et al. DexVerse: A Modular Benchmark for Multi-Task, Multi-Embodiment Dexterous Manipulation. arXiv:2607.08751, 2026.

[8] Z. Yang, Y. Zhang, D. Zhang, et al. Bench2Dex: Benchmarking Visuo-Tactile Bimanual Dexterous Manipulation Across Dexterous Hands. arXiv:2609.15726, 2026.

AuthorsOpenReview ↗

Response to Reviewer QKkY (3/7)

W1 (continued): Known generalization limits of VLAs and WAMs

High-level generalization is a widely reported weakness of current VLAs and WAMs. Independent studies find that policies which succeed on trained tasks drop sharply, often to near zero, on new tasks and on new combinations of skills they have already been shown:

StudyPoliciesTrained tasks or combinationsNew tasks or combinations
CRAFT [1]π0.5 [2] / π0 [3]100.0 / 99.7%0.7 / 0.3% on undemonstrated combinations of demonstrated skills
ACG-Bench [4]π0.5 on two arms37.0% in-domain2.9% on 17 unseen arm-wise compositions
LIBERO-PRO [5]VLAs on LIBEROover 90%0.0% under perturbed objects, initial states, instructions and environments
AGNOSTOS [6]π0 / OpenVLA [7]trained on diverse datasets17.5 / 13.6% on 23 unseen tasks
AdaWAM [8]FastWAM [9]trained LIBERO tasks0% on an unseen subtask combination

RT-2 [10] already observed that web-scale pretraining gives the robot no new motions, so its physical skills stay within those seen in the robot data. AGNOSTOS finds that current VLAs struggle on unseen tasks despite being trained on diverse datasets, and LIBERO-PRO traces the collapse to rote memorization of action sequences and layouts. For WAMs, cross-task generalization likewise remains difficult because predicted futures collapse on unseen tasks [11], and out of distribution, video generation models imitate the closest training case rather than the underlying rule [12]. π0.5 and FastWAM, both evaluated in DexOmni, are among the policies that fail in these studies. We have included this discussion in Appendix D.1 of revision.

Near-zero Transfer-16 success thus matches a documented limitation of these policy classes: it measures their limited ability to carry learned skills to new assets, relations and sequences, not a lack of data quality or diversity.

[1] T. Yang, Y. Na, Y. Cho, et al. Same Scene, Different Task: Skill Alignment for Compositional Generalization in VLAs. arXiv:2610.00524, 2026.

[2] Physical Intelligence, K. Black, N. Brown, et al. π0.5: a Vision-Language-Action Model with Open-World Generalization. arXiv:2504.16054, 2025.

[3] K. Black, N. Brown, D. Driess, et al. π0: A Vision-Language-Action Flow Model for General Robot Control. arXiv:2410.24164, 2024.

[4] Z. Zhang, B. Ran, Y. Wu, et al. Arm-wise Compositional Generalization in Dual-Arm Vision-Language-Action Models. arXiv:2610.06184, 2026.

[5] X. Zhou, Y. Xu, G. Tie, et al. LIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models Beyond Memorization. arXiv:2510.03827, 2025.

[6] J. Zhou, K. Ye, J. Liu, et al. Exploring the Limits of Vision-Language-Action Manipulations in Cross-task Generalization. arXiv:2505.15660, 2025.

[7] M. J. Kim, K. Pertsch, S. Karamcheti, et al. OpenVLA: An Open-Source Vision-Language-Action Model. CoRL 2024.

[8] Y. Tang, J. Xu, Y. Shang, et al. Dreaming when Necessary: Advancing World Action Models with Adaptive Multi-Modal Reasoning. arXiv:2606.07089, 2026.

[9] T. Yuan, Z. Dong, Y. Liu, H. Zhao. Fast-WAM: Do World Action Models Need Test-time Future Imagination? arXiv:2603.16666, 2026.

[10] A. Brohan, N. Brown, J. Carbajal, et al. RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control. CoRL 2023.

[11] C. Zhao, Z. Ye, Y. Lin, et al. What Limits Cross-Task Generalization in World Action Models? arXiv:2610.16641, 2026.

[12] B. Kang, Y. Yue, R. Lu, et al. How Far is Video Generation from World Model: A Physical Law Perspective. ICML 2025.

Reviewer_mboPOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DexOmni presents a simulation dataset and benchmark for dexterous manipulation. It provides a curated package of 18,000 human teleoperation demonstrations alongside a 50-task Core-50 evaluation catalog and 16 transfer tasks without target-task demonstrations in benchmark training. The package includes validation trajectories and recordings excluded from the evaluation catalog. Eight multitask checkpoints and a single-task Diffusion Policy reference are evaluated using native success predicates and diagnostics for activation, synchronization, goal retention, and stage ordering. The highest Core-50 task-macro success is 29.1%; Figure 1 reports 13 successful episodes among 6,232 valid Transfer-16 episodes. The paper analyzes skill-group differences, execution-time budgets, and failure transitions, and uses ordered progress to distinguish transfer behavior when final success is rare. Its contribution is the resource and the accompanying empirical characterization of the evaluated policies.

Critical strengths
  1. DexOmni expands human demonstration coverage and adds a defined task-transfer evaluation relative to DexJoCo. Transfer-16 includes skill combinations and deformable interactions without target-task demonstrations in benchmark training. Appendix B.13 acknowledges precedents for replay augmentation and task holdouts. The contribution is a larger dexterous resource combining these elements under an explicit training and evaluation protocol.

  2. The failure analysis identifies synchronization and retention problems hidden by aggregate success. In §5.5, both knobs reach their targets at some point in 219 of 800 Set Dual Knobs episodes, but simultaneously in only 38. In Open Laptop, OpenWAM and Spatial Forcing reach the target angle similarly often, yet retain it at their respective native episode endpoints in 67 and 23 of 100 episodes. These comparisons localize failures for the evaluated tasks and checkpoints under the reported stopping protocols.

Critical weaknesses
  1. Transfer-16 currently offers limited resolution for comparing complete-task generalization. Only 13 of 6,232 valid episodes succeed (Figure 1), leaving all evaluated checkpoints near the floor in final task success. Ordered progress and the failure diagnostics recover useful distinctions, but mainly characterize partial execution. This limits how much the current results establish about relative ability to complete new tasks. A more graduated range of transfer difficulty would help measure improvements in successful transfer while retaining the challenging cases.
Questions for authors
  1. Which tasks and semantic transitions account for the significant progress differences in §5.3? Please clarify whether the advantages are spread across tasks or concentrated in a few stages.

  2. Can simulator state independently validate the language probe's target assignments? What evidence distinguishes training-scene memorization from other scene-dependent preferences?

4 comments
AuthorsOpenReview ↗

Response to Reviewer mboP (1/3)

Thank you for your precise, constructive and positive review. We appreciate your recognition that DexOmni expands human-demonstration coverage with a defined task-transfer evaluation, and that the failure analysis identifies synchronization and retention problems hidden by aggregate success. Below, we address your concerns point by point.

W1: Graded resolution on Transfer-16

Ordered progress already gives Transfer-16 a graded range. For OpenWAM [1], per-task progress is 20–33% on five tasks (Cover Lunchbox, Set Mixer Knobs, Open Laptop and Click Mouse, Spray Shoe, Pack Pillow), 10–16% on four, and below 3% on six; Open Trash Bin (Gen02) logs no graded progress. Reporting Transfer-16 by these tiers measures progress on the easier tasks while retaining the challenging ones.

Complete success is also reachable within the time limits on some tasks. We evaluated two frontier models, GPT-6 and Fable 5.2, zero-shot on all 66 tasks, with 10 trials per task: neither receives DexOmni demonstrations, and each controls both hands through a single move_eef function that directly sets absolute palm poses and 20 finger-joint targets per hand, executed by the same native controllers, time limits and checkers and scored within 100 model responses per trial.

Suite / familyGPT-6 zero-shotFable 5.2 zero-shotBest DexOmni-trained checkpoint
Transfer-16, all 16 tasks1.9% (3/160)1.9% (3/160)0.9% (DM0.5 [2])
asset/relation (7 tasks)4.3% (3/70)4.3% (3/70)1.4% (DM0.5)
composition (6 tasks)0.0% (0/60)0.0% (0/60)0.7% (DM0.5)
deformable (3 tasks)0.0% (0/30)0.0% (0/30)0.0%
Core-50, all 50 tasks5.2% (26/500)5.2% (26/500)29.1% (OpenWAM)

GPT-6 completes Open Trash Bin (2/10) and Orient Bottle Upright (1/10), and Fable 5.2 completes Open Trash Bin (3/10), all single-stage asset-transfer tasks; over all 66 tasks they solve 12 and 14 without any demonstrations, which shows strong zero-shot task understanding. Neither solves a composition or deformable task, and each succeeds in only 2 of 180 trials on Core-50 functional tasks and in no tool-on-object task, because emitting high-dimensional palm-and-finger coordinates directly is still hard for them. Every Transfer-16 task is also verified in person by the operators (Appendix B.2); the tiers above measure the partial progress that precedes complete success. We have included the tiered reporting and the zero-shot results in Section 5.3 and Table 3 of revision.

[1] Y. Wang, S. Huang, M. Li, et al. OpenWAM: An Open, Modular Exploration Towards Systematic World-Action Model Pretraining. arXiv:2609.07398, 2026.

[2] E. Yu, H. Lv, J. Sun, et al. DM0: An Embodied-Native Vision-Language-Action Model towards Physical AI. arXiv:2602.14974, 2026.

AuthorsOpenReview ↗

Response to Reviewer mboP (2/3)

Q1: Where the progress differences in Section 5.3 come from

No single task carries the differences, which are concentrated in the first contact-dependent transition. The gaps over HY-Embodied and LingBot-VLA are spread over most tasks (14 and 13 tasks ahead), whereas three tasks (Spray Shoe, Cover Pitcher and Box, Place Fruit and Photograph Logo) give 36 of the 40 summed points of the gap over Spatial Forcing.

OpenWAM vs.Gap (points)Tasks ahead / behind / tiedGap after removing any one taskFrom the first transition
HY-Embodied [1]6.014 / 1 / 0 (sign test p = 0.001)4.9–6.54.8
LingBot-VLA [2]3.713 / 1 / 1 (p = 0.002)2.7–4.22.8
Spatial Forcing [3]2.69 / 2 / 4 (p = 0.07)2.0–2.92.3

Every leave-one-task-out interval excludes zero. The largest single contributions are moving the lunchbox lid (48/50 versus 18/50 for HY-Embodied), bringing the laptop screen to the requested angle (29/50 versus 5/49 for HY-Embodied and 10/50 for LingBot-VLA), and establishing the spray-bottle grasp (35/43 versus 14/43 for Spatial Forcing). Per task, the ordered progress of OpenWAM [4] and its difference from each checkpoint (points) are:

TaskOpenWAM progress (%)vs. HY-Embodiedvs. LingBot-VLAvs. Spatial Forcing
Cover Lunchbox33.3+21.3+0.7+0.7
Orient Bottle Upright2.0+1.0+2.00.0
Pass Glue Stick0.5+0.5+0.50.0
Set Mixer Knobs31.3+4.7+2.3−1.3
Spray Shoe20.4+2.3+2.3+12.2
Stack Boxes1.4+1.4+1.4+1.4
Cover Pitcher and Box16.0+6.4+6.0+12.0
Open Laptop and Click Mouse27.5+22.4+18.0+2.0
Pass Block and Close Box0.3+0.3+0.3+0.3
Place Fruit and Photograph Logo15.0+11.0+15.0+12.0
Scrub Bottle Then Fill Cup14.3+5.7+8.30.0
Season and Stir1.6+1.2+1.6+1.2
Compress Sponge into Mug10.6+4.6−3.7+0.6
Fold Towel Twice0.0−0.50.0−1.5
Pack Pillow20.0+8.0+0.40.0

We have included the per-task and per-transition table in Appendix D of revision.

[1] H. Zhang, L. Xiang, H. Lin, et al. Hy-Embodied-0.5-VLA: From Vision-Language-Action Models to a Real-World Robot Learning Stack. arXiv:2606.14409, 2026.

[2] W. Wu, F. Lu, Y. Wang, et al. A Pragmatic VLA Foundation Model. arXiv:2601.18692, 2026.

[3] F. Li, W. Song, H. Zhao, et al. Spatial Forcing: Implicit Spatial Representation Alignment for Vision-language-action Model. arXiv:2510.12276, 2025.

[4] Y. Wang, S. Huang, M. Li, et al. OpenWAM: An Open, Modular Exploration Towards Systematic World-Action Model Pretraining. arXiv:2609.07398, 2026.

AuthorsOpenReview ↗

Response to Reviewer mboP (3/3)

Q2: Validating the language probe with simulator state; memorization versus other preferences

Yes. For every paper episode of the three object-selection tasks (100 per checkpoint and task), we read the object the policy actually moved from the simulator's start and end object poses. We compare how often it was the requested object with a permutation baseline: the rate a policy would reach if it kept its own object preferences but ignored the request. Each cell shows the match rate in % over the episodes with a choice (the object displaced most, by more than 3 cm; for Largest Fitting Ball, the first ball touched with more than 1 N), 85–100 of 100 per cell except OpenWAM and FastWAM on Largest Fitting Ball (49 and 65), with the baseline in parentheses.

CheckpointBall in BowlFruit in BucketLargest Fitting Ball
π0.5 [1]85.6 (39.2)75.5 (25.1)80.6 (34.8)
DM0.5 [2]88.2 (38.0)86.7 (25.7)81.5 (35.5)
OpenWAM [3]86.5 (36.8)81.6 (25.6)89.8 (34.3)
FastWAM [4]92.5 (34.6)85.6 (25.5)83.1 (34.6)
Motus [5]86.5 (35.2)93.9 (25.7)78.8 (34.5)
HY-Embodied [6]43.8 (38.3)29.9 (25.2)33.7 (34.2)
LingBot-VLA [7]35.8 (38.2)29.3 (24.8)34.8 (34.2)
Spatial Forcing [8]39.6 (35.7)39.0 (24.5)29.7 (33.5)

Five checkpoints follow the request far above their baselines (p ≤ 0.0002 on every task), while HY-Embodied and LingBot-VLA match only their baseline (p ≥ 0.16), as does Spatial Forcing on two of the three tasks. New rollouts with a per-step pose and contact logger agree: the first object lifted is the requested one in 35–43 of 50 episodes for π0.5 and DM0.5, but in only 5–14 of 50 for HY-Embodied and LingBot-VLA. Simulator state therefore independently confirms the probe's finding of which checkpoints follow the request.

For the second question, we tested four families of explanations of the choices made by the three checkpoints that ignore the request, against the training split: the target of the nearest training demonstration by initial layout (memorization), a fixed side, a fixed object identity, and proximity to the palm. Each is tested against two permutation nulls with a Bonferroni correction within each cell.

  • Layout memorization fits one of nine checkpoint–task cells: HY-Embodied on Ball in Bowl, where evaluation layouts lie within 1.6 cm of training layouts per object (median).
  • Other patterns are fixed preferences: in three cells, LingBot-VLA favours the pear, HY-Embodied the baseball, and Spatial Forcing the object nearest its right palm. In the remaining five cells no pattern beats chance.
  • The test does not over-fire: for the five request-following checkpoints it finds no pattern (11 nominal hits in 160 tests, about the 8 expected by chance).

Memorization of the nearest training layout therefore explains little of the request-ignoring behaviour; where a pattern exists, it is mostly a fixed object or reach preference. We have included these analyses in Appendix D of revision. We hope these results address your questions, and we would be glad to discuss any remaining ones.

[1] Physical Intelligence, K. Black, N. Brown, et al. π0.5: a Vision-Language-Action Model with Open-World Generalization. arXiv:2504.16054, 2025.

[2] E. Yu, H. Lv, J. Sun, et al. DM0: An Embodied-Native Vision-Language-Action Model towards Physical AI. arXiv:2602.14974, 2026.

[3] Y. Wang, S. Huang, M. Li, et al. OpenWAM: An Open, Modular Exploration Towards Systematic World-Action Model Pretraining. arXiv:2609.07398, 2026.

[4] T. Yuan, Z. Dong, Y. Liu, H. Zhao. Fast-WAM: Do World Action Models Need Test-time Future Imagination? arXiv:2603.16666, 2026.

[5] H. Bi, H. Tan, S. Xie, et al. Motus: A Unified Latent Action World Model. arXiv:2512.13030, 2025.

[6] H. Zhang, L. Xiang, H. Lin, et al. Hy-Embodied-0.5-VLA: From Vision-Language-Action Models to a Real-World Robot Learning Stack. arXiv:2606.14409, 2026.

[7] W. Wu, F. Lu, Y. Wang, et al. A Pragmatic VLA Foundation Model. arXiv:2601.18692, 2026.

[8] F. Li, W. Song, H. Zhao, et al. Spatial Forcing: Implicit Spatial Representation Alignment for Vision-language-action Model. arXiv:2510.12276, 2025.

AuthorsOpenReview ↗

Gentle Follow-up to Reviewer mboP

Dear Reviewer mboP,

Thank you again for your thoughtful review. As the discussion period closes on November 18, and we fully understand how busy this time must be for you, please allow us to briefly summarize our response in case it is helpful.

Following your comments, we (1) showed that ordered progress gives Transfer-16 a graded range even though final success is near zero: the best checkpoint reaches 20–33% progress on five tasks, 10–16% on four and below 3% on six; (2) located the significant progress differences of Section 5.3, as you asked: the gaps of the best checkpoint over two of the three others are spread over most tasks (ahead on 14 and 13 of 15), three tasks carry 36 of the 40 summed points of the gap over the third, and most of each gap comes from the first contact-dependent transition, for example moving the lunchbox lid (48 versus 18 of 50 episodes); (3) validated the language probe with simulator state: over 100 evaluated episodes per checkpoint and task, five checkpoints choose the requested object far above their permutation baseline (p ≤ 0.0002 on every task) while the two request-ignoring VLAs match only their baseline, and new rollouts with a pose and contact logger agree (the first object lifted is the requested one in 35–43 of 50 episodes for two request-following checkpoints, against 5–14 of 50 for the two request-ignoring ones); and (4) separated training-layout memorization from other scene-dependent preferences: memorization fits only one of nine checkpoint–task cells, three others follow fixed object or palm-proximity preferences, and for the request-following checkpoints the test finds no pattern (11 nominal hits in 160 tests, about the 8 expected by chance). All changes are highlighted in blue in the revised PDF (Section 5.3 and Appendix D).

We would greatly value your thoughts on whether these additions address your concerns, and we would be glad to clarify anything further in the time remaining. Thank you again for your time and effort.

Best regards, The Authors

Reviewer_9H8wOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

DexOmni provides 18,000 curated human demonstrations and a simulation benchmark for dexterous manipulation. Core-50 contains training and execution tasks; Transfer-16 contains evaluation-only tasks involving changed assets or relations, new skill compositions, and deformable objects. The paper evaluates a single-task diffusion-policy reference and eight multitask VLA or world-action checkpoints. The best Core-50 score is 29.1%, while Transfer-16 is almost entirely unsolved. Analyses of task stages, time budgets, language conditioning, and demonstration counts examine where the policies fail.

Critical strengths
  1. The evaluation protocol is documented carefully enough to make the results interpretable. The paper specifies native success conditions and physical-time budgets, separates policy failures from invalid runs, and reports uncertainty with its resampling assumptions. Appendix B.2 describes human verification of task feasibility and independent demonstration review. The authors also disclose the DP sampler mismatch, limits of pretrained-data provenance, and single-seed scaling setup. For a benchmark paper, this attention to what each score actually measures is a major strength.
  2. The failure analysis is the most valuable part of the paper. The mouse and dual-knob examples distinguish acquiring an object from actuating it at the right time; the laptop and handover examples show failures to maintain a reached goal or release only after support transfers. These observations suggest concrete problems to work on. The time-budget analysis is also revealing: twelve of twenty-eight checkpoint pairs reverse order between the 1x and 3x expert-duration budgets. The dataset and these analyses together support acceptance, even though the baseline comparisons have limitations.
Critical weaknesses
  1. The leaderboard is still a comparison of selected checkpoints under different training and deployment choices. Pretraining, fine-tuning schedules, and replanning cadence differ, and the rollout intervals do not include training-run or checkpoint-selection variability. Most DP evaluations also use an earlier goal sampler. The paper acknowledges these issues, so I do not read the rankings as architectural conclusions. Still, repeated training and matched-cadence comparisons on a small task subset would make the main performance differences more convincing.
  2. Transfer-16 is so difficult that final success provides little separation between methods: there are thirteen successes in 6,232 valid episodes. Ordered progress helps, and human verification rules out simple task infeasibility, but several tasks change multiple factors together. It remains difficult to distinguish weak source skills from the extra difficulty of transfer. Matched intermediate variants would improve this diagnosis. The held-out visual and paraphrase protocols in Appendix B.10 also need dedicated results; the reported in-distribution invariance checks do not answer those questions.
Questions for authors
  1. Are the main checkpoint differences stable across training seeds and matched replanning cadences on a representative task subset? Can the DP reference be scored with the current goal sampler?
  2. For Transfer-16, can source-task competence be related to conditional transition success in the target task? Are there matched variants that help separate object changes from composition difficulty?
  3. Are paired held-out paraphrase and visual-shift results available under the protocol in Appendix B.10? They would help establish the benchmark’s coverage beyond the task-transfer results.
6 comments
AuthorsOpenReview ↗

Response to Reviewer 9H8w (1/5)

Thank you for the constructive review and for recognizing the careful evaluation protocol and concrete failure analysis. We address your concerns below.

W1: Training and deployment choices behind the leaderboard

We agree that the leaderboard compares training-and-deployment recipes rather than isolated architectural choices, and Section 5.1 now states this explicitly.

Replanning cadence is a deployment choice that differs across checkpoints and can be changed without retraining. In the paper, each checkpoint executed its integration's default number of actions per plan: 30 for π0.5 [1], DM0.5 [2], HY-Embodied [3] and LingBot-VLA [4], and 16 for OpenWAM [5], FastWAM [6], Motus [7] and Spatial Forcing [8]. To separate cadence from the policy, we re-evaluated all eight checkpoints at both 16 and 30 executed actions per plan, with unchanged weights, on an 11-task block with the same 50 draws per task and the native checkers. The block holds the twelve Core-50 tasks that separate the checkpoints most clearly relative to binomial noise in the paper results, minus Lift Carton, which runs on a separate force-logging route; it was fixed before any new run and averages 33.7% in those results. The table reports task-macro success in % (points for differences).

Checkpoint (paper cadence)30 steps16 steps16 − 30 (95% CI)Re-run − paper
π0.5 (30)44.453.3+8.9 (+4.0, +13.6)+1.3
DM0.5 (30)37.344.2+6.9 (+2.7, +11.1)+2.0
HY-Embodied (30)26.929.3+2.4 (−2.0, +6.9)+0.7
LingBot-VLA (30)32.734.2+1.5 (−2.9, +5.8)+0.4
OpenWAM (16)53.546.4−7.1 (−12.0, −2.2)+0.5
FastWAM (16)36.738.0+1.3 (−3.5, +5.8)−0.7
Motus (16)32.429.1−3.3 (−7.5, +0.9)−1.1
Spatial Forcing (16)11.612.9+1.3 (−2.2, +4.7)+0.7

Cadence is a significant deployment factor, and its effect is model-specific: π0.5 and DM0.5 gain at 16 steps, OpenWAM gains at 30, and the other five change by less than 3.5 points with intervals that include zero. Matching cadence changes 3 of the 28 pairwise orders at 16 steps and 1 at 30, and with each checkpoint at its better cadence the top two, OpenWAM (53.5) and π0.5 (53.3), are within 0.2 points. The last column re-runs each checkpoint at its paper cadence on the same draws: the fresh runs reproduce the paper's success to within −1.1 to +2.0 points, and every interval includes zero.

Independent training seeds would indeed be a direct measure of training-run variability. Each multi-task checkpoint, however, was trained for 70,000–151,000 updates on multi-GPU clusters, so retraining all eight with additional seeds exceeds the compute we can secure within the discussion period. As the closest measure within that budget, we evaluated the adjacent saved checkpoint of the six checkpoints that have one (1,000–5,000 updates earlier; π0.5 and OpenWAM keep only the evaluated checkpoint) on the same block, draws and cadence. This measures the variation from checkpoint selection within a run, not across training runs; LingBot-VLA's evaluated 151,000 continues its 150,000 checkpoint for 1,000 updates at batch 1,216:

CheckpointEvaluatedAdjacentAdjacent − evaluated (95% CI)
DM0.537.334.9−2.4 (−6.5, +1.8)
HY-Embodied26.924.7−2.2 (−5.6, +1.5)
LingBot-VLA32.733.3+0.5 (−2.9, +4.0)
Spatial Forcing12.910.5−2.4 (−5.5, +0.7)
FastWAM38.038.5+0.5 (−3.6, +4.7)
Motus29.130.2+1.1 (−2.7, +4.9)

Every difference lies within −2.4 to +1.1 points with an interval that includes zero, and none of the 28 pairwise orders changes.

We have included these results in Section 5.1 and Appendix C of revision.

[1] Physical Intelligence, K. Black, N. Brown, et al. π0.5: a Vision-Language-Action Model with Open-World Generalization. arXiv:2504.16054, 2025.

[2] E. Yu, H. Lv, J. Sun, et al. DM0: An Embodied-Native Vision-Language-Action Model towards Physical AI. arXiv:2602.14974, 2026.

[3] H. Zhang, L. Xiang, H. Lin, et al. Hy-Embodied-0.5-VLA: From Vision-Language-Action Models to a Real-World Robot Learning Stack. arXiv:2606.14409, 2026.

[4] W. Wu, F. Lu, Y. Wang, et al. A Pragmatic VLA Foundation Model. arXiv:2601.18692, 2026.

[5] Y. Wang, S. Huang, M. Li, et al. OpenWAM: An Open, Modular Exploration Towards Systematic World-Action Model Pretraining. arXiv:2609.07398, 2026.

[6] T. Yuan, Z. Dong, Y. Liu, H. Zhao. Fast-WAM: Do World Action Models Need Test-time Future Imagination? arXiv:2603.16666, 2026.

[7] H. Bi, H. Tan, S. Xie, et al. Motus: A Unified Latent Action World Model. arXiv:2512.13030, 2025.

[8] F. Li, W. Song, H. Zhao, et al. Spatial Forcing: Implicit Spatial Representation Alignment for Vision-language-action Model. arXiv:2510.12276, 2025.

AuthorsOpenReview ↗

Response to Reviewer 9H8w (2/5)

W2: Separating checkpoints on Transfer-16

Ordered progress separates checkpoints on Transfer-16: OpenWAM [1] exceeds HY-Embodied [2], LingBot-VLA [3] and Spatial Forcing [4] by 6.0, 3.7 and 2.6 points, with task-bootstrap intervals that exclude zero, and no single task carries the advantage:

OpenWAM vs.Progress gap (points)Tasks ahead / behind / tiedGap with any one task removedGap from the first transition
HY-Embodied6.014 / 1 / 0 (sign test p = 0.001)4.9–6.54.8
LingBot-VLA3.713 / 1 / 1 (p = 0.002)2.7–4.22.8
Spatial Forcing2.69 / 2 / 4 (p = 0.07)2.0–2.92.3

Most of each gap comes from the first contact-dependent transition, for example moving the lunchbox lid (48/50 episodes for OpenWAM versus 18/50 for HY-Embodied) and bringing the laptop screen to the requested angle (29/50 versus 5/49 for HY-Embodied and 10/50 for LingBot-VLA).

To tell weak source skills from the difficulty of transfer, we related each target transition to the same transition in its source task (Q2 below). To test whether the tasks can be solved without DexOmni data, we evaluated two frontier models, GPT-6 and Fable 5.2, zero-shot on all 66 tasks, with 10 trials per task. Neither receives DexOmni demonstrations or finetuning; each observes three RGB views, the measured robot state and the instruction, and controls both hands through a single move_eef function: each call directly sets absolute palm poses and 20 finger-joint targets per hand, a 52-dimensional command that the same native controllers execute at 30 Hz, with no grasp primitive. Both are scored within 100 model responses per trial.

Suite / interaction typeGPT-6 zero-shotFable 5.2 zero-shotBest DexOmni-trained checkpoint
Core-50, all 50 tasks5.2% (26/500)5.2% (26/500)29.1% (OpenWAM)
Core-50, primitive (17)7.6% (13/170)11.2% (19/170)42.7% (FastWAM [5])
Core-50, functional (18)1.1% (2/180)1.1% (2/180)23.8% (OpenWAM)
Core-50, contact-rich (15)7.3% (11/150)3.3% (5/150)22.1% (OpenWAM)
Transfer-161.9% (3/160)1.9% (3/160)0.9% (DM0.5 [6])

The equal totals come from different successes: both models run the same 660 fixed trials, yet only 6 of the 29 successes of each fall on the same trial, close to the 6.6 expected if they succeeded independently on each task, and their success counts differ on all 19 tasks that either model solves. The results show two things. First, neither model is yet good at emitting high-dimensional dexterous coordinates directly. GPT-6's successes concentrate where one coarse palm motion reaches the goal: spinning the propeller (7/10), lifting the carton with both hands (4/10), turning a cup upright (4/10) and opening the micropipette box (3/10). Each model succeeds in only 2 of 180 functional-task trials and in no tool-on-object task, where individual fingers must actuate a held tool, and neither solves a composition or deformable Transfer-16 task. Second, their zero-shot ability is strong: with no demonstrations GPT-6 solves 12 of the 66 tasks and Fable 5.2 solves 14, only 7 of them in common, so 19 tasks across all three Core-50 interaction types and two Transfer-16 tasks (Open Trash Bin, Orient Bottle Upright) are solved by at least one model, and GPT-6's saved action notes state prerequisites and monitor execution, e.g., "neither object has yet visibly moved, so do not lift or click yet" in Open Laptop and Click Mouse. With every task also verified in person by the operators (Appendix B.2), the tasks are attainable, and their difficulty lies in high-dimensional dexterous control rather than in task understanding or feasibility, which is also where the trained policies fail.

The paired Appendix B.10 results are in Q3. We have included the GPT-6 and Fable 5.2 references and the per-task breakdown in Section 5.3 and Appendix D of revision.

[1] Y. Wang, S. Huang, M. Li, et al. OpenWAM: An Open, Modular Exploration Towards Systematic World-Action Model Pretraining. arXiv:2609.07398, 2026.

[2] H. Zhang, L. Xiang, H. Lin, et al. Hy-Embodied-0.5-VLA: From Vision-Language-Action Models to a Real-World Robot Learning Stack. arXiv:2606.14409, 2026.

[3] W. Wu, F. Lu, Y. Wang, et al. A Pragmatic VLA Foundation Model. arXiv:2601.18692, 2026.

[4] F. Li, W. Song, H. Zhao, et al. Spatial Forcing: Implicit Spatial Representation Alignment for Vision-language-action Model. arXiv:2510.12276, 2025.

[5] T. Yuan, Z. Dong, Y. Liu, H. Zhao. Fast-WAM: Do World Action Models Need Test-time Future Imagination? arXiv:2603.16666, 2026.

[6] E. Yu, H. Lv, J. Sun, et al. DM0: An Embodied-Native Vision-Language-Action Model towards Physical AI. arXiv:2602.14974, 2026.

AuthorsOpenReview ↗

Response to Reviewer 9H8w (3/5)

Q1: Stability of the main differences; DP under the current goal sampler

Stability across replanning cadences and checkpoints is shown in W1: matching cadence changes at most 3 of the 28 pairwise orders, and the adjacent saved checkpoints, available for six of the eight, change none. For DP [1], we re-evaluated the 38 tasks whose evaluations had used the earlier goal sampler, now under the current sampler and on the same draws as π0.5 (seeds, goals and prompts), so all 50 DP cells now share π0.5's draws:

DP (single-task)Successes / episodesSR (%)95% CI, episodes95% CI, tasks + episodesTasks ≥ 50%
Submission483 / 5,0009.79.0–10.35.0–14.93
Revision506 / 5,00010.19.5–10.85.3–15.63

DP remains the single-task reference below all eight multi-task checkpoints (12.4–29.1%). We have updated Table 2 and the per-task table in Appendix C of revision.

Q2: Source-task competence versus transition success; matched variants

Yes. We measured the same native checker transition in each Transfer-16 task and in the Core-50 task it draws on, using the stage traces of π0.5 [2] and OpenWAM [3]. In 28 of 33 comparable transitions the target rate is lower than the source rate, although the transitions keep roughly the same order of difficulty (Spearman 0.69). The drop is large even where the skill itself is well learned:

Transition (source → target)π0.5OpenWAM
Left-hand pickup (Pass Block → Pass Glue Stick)72/100 → 0/5088/100 → 1/50
First seasoning on target (Pour Seasonings → Season and Stir)83/100 → 0/5097/100 → 4/50
Safe brush grasp (Scrub Bottle → Scrub Bottle Then Fill Cup)67/95 → 0/4971/100 → 0/50

Across the eight checkpoints, source-task success predicts conditional transition success within compositions (mean within-transition Spearman 0.32, 95% CI 0.05–0.64). For matched variants, the Gen tasks mainly change assets or relations, while the Comb tasks mainly compose skills that are already learned, so the family split largely separates object change from composition.

Near-zero Transfer-16 success therefore reflects the policies' failure to carry learned skills to new assets, relations and sequences, which is what Transfer-16 measures. We have included this comparison in Section 5.3 and Appendix D.1 of revision, with the matched transitions in Section 5.5 and Figure 4(c).

[1] C. Chi, Z. Xu, S. Feng, et al. Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. RSS 2023.

[2] Physical Intelligence, K. Black, N. Brown, et al. π0.5: a Vision-Language-Action Model with Open-World Generalization. arXiv:2504.16054, 2025.

[3] Y. Wang, S. Huang, M. Li, et al. OpenWAM: An Open, Modular Exploration Towards Systematic World-Action Model Pretraining. arXiv:2609.07398, 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.