acceptodds
Withdrawn from ICLR 2027

DexWorld-Bench: Evaluating Fine-Grained Control and Interaction for Dexterous World Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

World models for dexterous manipulation have served as learned simulators for anticipating action consequences and supporting robotic planning and policy learning. Their utility depends on faithfully capturing fine-grained finger motions and hand-object interactions, where small control differences can change manipulation outcomes. However, existing evaluation methods provide limited systematic coverage of these dynamics. Similar overall motion or successful task completion can conceal errors in local control and interaction. We introduce DexWorld-Bench, a benchmark that evaluates dexterous world models along three complementary dimensions: fine-grained control accuracy, interaction consistency, and task executability. The benchmark comprises 220 full task sequences, 90 critical interaction clips that include both successful and failed interactions, and 70 controlled action comparisons that isolate the motion of a single arm or finger. We evaluate multiple widely-used world models spanning general-purpose video generators, action-conditioned world models, and world action models under the same protocol. The results show that action conditioning alone does not ensure that models faithfully follow fine-grained action commands, and that fine-grained control and long-horizon executability remain key bottlenecks in current dexterous world models.

Primary area:
datasets and benchmarks
Keywords:
Benchmark, Dexterous hand, World Model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_zmw8OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper designed a dedicated benchmark for evaluating video generation models (text-conditioned, action-conditioned, WAMs) on DexDoJo dataset, benchmarking video generation qualities of hand and arm details, object trajectories, and executabilities for these models on simulated dexterous manipulation context.

Critical strengths
  1. This paper did a good job in doing controlled and rigorous benchmarking experiments on various video generation models and provided sufficient experimental evidence with illustrative figures. This is helpful for reproducing similar patterns and isolating different video generation features from different models.
  2. The benchmark design is well verified with calibration efforts. Ground-truth replay tests, comparisons between MLLM and human judgments, and IDM validation tests provide useful checks on evaluator reliability. The benchmark also includes failure interactions which rewards faithful generation of failed trajectories.
Critical weaknesses
  1. Modest technical contribution. The paper only benchmarks several video generation models, simply coming up with some heuristic metrics for comparison, without proposing promising and constructive alternatives for improvement. This makes this work simply a blog post for cage fighting existing methods.
  2. Executability is bounded by IDM. In actual manipulation replay setting, the IDM is trained with a very small scale of DexDoJo trajectories, so essentially the IDM is overfitting on these trajectories, which makes it extremely vulnerable to OOD scenarios. Even so, the IDM achieves only 0.87 success on ground-truth videos. Therefore, the IDM will easily hallucinate when the generated videos are slightly off the distribution of clean simulator images, making the executability benchmarking essentially 'good-looking' benchmarking. Moreover, inferring 16 actions from two endpoint images is too ambiguous for contact-rich motion.
  3. Dynamic, fine-grained and contact-rich motion. The key challenges in dexterous manipulation are (1) the high-dynamics of high-DoF robot systems, where close loop reactivity is crucial, and (2) ubiquitous contact and contact switching. Unfortunately, this work addresses none of these: the motions are mostly quasi-static open-loop action chunk replay without high dynamics and closed-loop feedback, and the contacts are mostly power grasps, without delicate finger gait switching or in-hand reorientation, and this benchmark doesn't even benchmark contact. Therefore, the benchmark might be informative for video generation on this specific domain, but will not be interesting to people actually working on real-world dexterous hands. It's also doubtful that the nuanced contact can be recovered by visual only input, where vision itself is not sufficient anymore under limited resolutions and heavy occlusions.
  4. Limited data scale. The benchmark only contains 220 full task sequences, which is very small. This scale of data can be easily overfitted by video generation models so it cannot establish correlation reliably. Also the generalization coverage is narrow since there is only one simulator and one embodiment, and no actual held-out tasks. Separate held-out results and tests across new tasks, objects, appearances, or embodiments would better support broad conclusions about these models.
  5. No real world experiments. Similar to 3, this work simply benchmarks video generation in a simulated domain, without real robot experiments for validation.
Questions for authors

Could authors provide more details on evaluation settings and protocols, e.g., fine-tuning budgets, IDM training, checkpoint selection, and generation settings?

1 comment

We thank the reviewer for the constructive feedback and address the main concerns below.

  1. Technical contribution (W1). Our contribution is a systematic evaluation framework for world models for dexterous manipulation. In particular, we introduce a controlled, finger-level action evaluation protocol to test whether models respond to actions prescribed for a specific finger, rather than merely reproducing whole-hand motion. The framework also jointly evaluates interaction outcomes and task executability. Our findings suggest concrete directions for improvement: strengthening finger-level data augmentation and structured action conditioning, improving state preservation and limiting error accumulation over long rollouts, and addressing hand deformations in generated videos to improve the accuracy of finger structure and pose prediction.
  2. Executability (W2, W5). We broaden the IDM training distribution with original demonstrations, randomized data, successful and failed policy rollouts, and self-play to improve robustness and generalization. Our two-frame-conditioned action decoding follows GR00T N1[1] and DreamGen[2], which predict action sequences between current and future frames. To address ambiguity over longer horizons, we will evaluate a shorter, six-step action chunk in the revision: at 5 fps and a 30 Hz control frequency, consecutive video frames span 0.2 seconds, corresponding to six control steps. This aligns the decoding horizon with the video sampling interval. Our Task Executability evaluation assesses generated videos for downstream execution, with the IDM serving as a fixed action decoder. The resulting Execution Success Rate therefore measures the performance of the combined video-model–decoder system. We will also add real-robot evaluations in the revised version.
  3. Dynamic manipulation, contact, and visual observations (W3). The benchmark’s tasks extend beyond power grasps: assembly, button pressing, and pump-handle operation require contact at specific locations, while the task suite also includes dynamic manipulation, tool use, and bimanual coordination. Our Interaction Outcome Consistency metric further examines whether these actions produce the corresponding observable consequences. For example, in grasping tasks, it distinguishes fingers closing without moving the object, dragging the object, and lifting it before it falls back onto the table. Future revisions will incorporate in-hand manipulation tasks, multi-view evaluation, and hand–object contact assessment to expand coverage of contact switching and mitigate observation limitations caused by visual occlusion.
  4. Data scale (W4). The 220 full task sequences constitute an evaluation subset selected from DexJoCo’s 1,100 demonstrations; they are not the models’ entire training dataset. We acknowledge that the current evaluation primarily covers different episodes within the same task categories, and that using a single simulator and embodiment limits the scope of our generalization claims. We will clarify this scope in the revision and extend future evaluations to include generalization tests across new tasks, objects, and appearance variations.
  5. Baseline configurations (Q1). Each baseline was trained using the configurations provided in its corresponding repository for approximately 100 epochs, by which point the training loss had stabilized. The conditioning inputs and output views used for video generation are summarized below.
BaselineConditioning inputs used for video generationOutput views
WanMultiview first frames + instruction; no actions2 views for single-arm tasks; 3 for bimanual tasks
IRASimMultiview first frames + instruction + actions2 views for single-arm tasks; 3 for bimanual tasks
Ctrl-WorldMultiview first frames + instruction + actions2 views for single-arm tasks; 3 for bimanual tasks
FastWAMPrimary-view first frame + instruction; no actionsPrimary view only
Cosmos3Primary-view first frame + instruction + actionsPrimary view only

[1] Bjorck, Johan, et al. "Gr00t n1: An open foundation model for generalist humanoid robots." arXiv preprint arXiv:2503.14734 (2025).

[2] Jang, Joel, et al. "Dreamgen: Unlocking generalization in robot learning through video world models." arXiv preprint arXiv:2505.12705 (2025).

Reviewer_iwAbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces DexWorld-Bench, which evaluates fine-grained action response, interaction consistency, and task executability using data from 11 DexJoCo tasks. Experiments with five models, including execution evaluation on three tasks, suggest that visual quality can conceal control and interaction errors.

Critical strengths

Matched arm interventions and isolated-finger commands directly probe whether models respond to altered actions, addressing a meaningful limitation of aggregate video metrics.

Including failed interactions appropriately distinguishes predictive fidelity from task success. Ground-truth calibration and human validation of interaction-outcome scores also provide useful checks on the evaluators.

Critical weaknesses

The reliability of several core metrics remains insufficiently established. The finger judge is calibrated on reference clips, while the reported IDM replay calibration covers only training-set sequences (Tables 3–4), leaving evaluator generalization insufficiently controlled. Appendix A.2 also assigns zero to complete tracking loss while permitting negative object trajectory scores, so object disappearance can improve the score.

The action-conditioned comparison is underspecified. Section 4.1 omits baseline-specific conditioning and inference details; notably, the original Fast-WAM removes future-video generation at inference, making the evaluated variant or adaptation essential to understanding this comparison (Yuan et al., 2026a, Secs. 3.2–3.3). Without establishing how prescribed actions condition its generated videos, the reported 0.50 arm-control accuracy cannot be confidently interpreted as failure to follow those actions.

Questions for authors

Which FastWAM variant is evaluated, and how are prescribed action sequences injected into each world-action model? Please provide the corresponding fine-tuning and video-generation configurations.

Do human assessments of generated finger motions confirm the reported control accuracies, and do execution gaps persist against GT-IDM replay on the same held-out episodes? Please report training and held-out execution results separately, and clarify the relationship between the 20 finger comparisons in Sec. 4.1 and the 24 scored clips plus eight controls in Appendix A.4.

How do OTS rankings change after correcting the treatment of complete tracking loss so that disappearance cannot improve a negative score? Please also define the frame-0 reproduction check and report the fraction of excluded cases for each model.

1 comment

We thank the reviewer for the constructive feedback and address the main concerns below.

  1. Reliability of the evaluation metrics (W1, Q2, Q3)
    1. OTS. No cases of complete object tracking loss occurred in our evaluation. Therefore, changing only the score assigned to this branch would leave the reported scores and rankings unchanged. The first-frame reproduction check computes the Pearson correlation coefficient between the RGB pixel values of the generated and reference first frames at the same resolution. Samples with a correlation below 0.95 are excluded from OTS evaluation. For Wan, 30/90 ID samples (33.3%) and 30/70 OOD samples (42.9%) were excluded. No samples from the other models were excluded by this check.
    2. IDM. We broadened the IDM training distribution using original demonstrations, randomized data, successful and failed policy rollouts, and self-play, and used multitask pretraining followed by task-specific fine-tuning to improve action recovery accuracy. The real-video calibration results reported in the paper show an end-effector position error of 1.05 cm, a rotation error of 0.048 rad, and a finger joint error of 0.020 rad, indicating that the IDM can recover reference actions with relatively high accuracy. However, low action reconstruction error does not guarantee successful replay: hand occlusions make some joint states difficult to infer, and local prediction errors can accumulate during open-loop execution, disrupting precise contact interactions. Our executability evaluation assesses the downstream utility of video models for execution, with the IDM serving as an action decoder (action expert). The resulting success rate therefore measures the performance of the execution system comprising the video model and a fixed action decoder.
    3. Finger control. The 24 clips in the appendix are the selected reference intervention clips across tasks. The generated videos corresponding to the 20 reference clips correctly identified by the judge are used to compute the main-table scores. The eight control clips are original manipulation videos without interventions; they are evaluated separately and are excluded from the scored set. We will add results from a blinded human evaluation in the revised version.
  2. Baseline configurations (W2, Q1). FastWAM was trained with paired video–action supervision, but the prescribed actions were not provided during video generation. We will correct this in the revised version. Each baseline was trained using the configurations provided in its corresponding repository for approximately 100 epochs, by which point the training loss had stabilized. The conditioning inputs and output views used for video generation are summarized below.
BaselineConditioning inputs used for video generationOutput views
WanMultiview first frames + instruction; no actions2 views for single-arm tasks; 3 for bimanual tasks
IRASimMultiview first frames + instruction + actions2 views for single-arm tasks; 3 for bimanual tasks
Ctrl-WorldMultiview first frames + instruction + actions2 views for single-arm tasks; 3 for bimanual tasks
FastWAMPrimary-view first frame + instruction; no actionsPrimary view only
Cosmos3Primary-view first frame + instruction + actionsPrimary view only
Reviewer_bxUEOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces DexWorld-Bench, a benchmark for evaluating video-generative world models for dexterous robotic manipulation. It evaluates (1) fine-grained control accuracy, (2) interaction consistency, and (3) task executability, assessed by recovering actions from generated videos and replaying them in simulation. The benchmark contains 220 full-task sequences, 90 critical interaction clips, and 70 controlled action comparisons across 11 simulated manipulation tasks. Evaluations of five representative models reveal substantial limitations in fine-grained action conditioning and long-horizon prediction fidelity.

Critical strengths
  • Relevant problem formulation: evaluating action-conditioned world models beyond conventional visual quality metrics is an important problem, and relatively under-studies for dexterous hand manipulation

  • Fine-grained evaluation: good comparison results that isolate individual finger or arm movements, a useful way to measure whether generated videos actually respond to prescribed actions. Also has multiple complementary eval metrics, including failed interactions is valuable for evaluating predictive fidelity rather than simply task success.

  • Baseline coverage: five representative models, including general-purpose video generation, action-conditioned world models, and world action models, under a shared evaluation protocol.

Critical weaknesses
  • Limited conceptual novelty. The benchmark largely extends existing evaluation protocols to dexterous hands. Many proposed metrics such as simulation executability are not unique to dexterous hands. The key distinction from prior benchmarks beyond finger-level control evaluation is insufficiently established.

  • Insufficient evaluation of complex contact dynamics. Despite emphasizing the complexity of dexterous manipulation, the benchmark mainly measure whether individual fingers or arms move as commanded, and do not systematically investigate contact dynamics, friction, slipping, or how subtle changes in finger actions affect physical interaction outcomes.

  • The above is also a result of limited simulation realism. All evaluations use DexJoCo's rigid-body simulation with relatively simplistic renderings. The benchmark does not capture compliant, deformable objects or realistic visual complexity. It seems like the held-out evaluations use new episodes of the same training tasks, and not testing novel tasks or real-world transfer.

  • Questionable task-executability evaluation. The proposed metric uses an IDM to recover actions from generated videos and replays them open-loop in simulation. This introduces errors from the IDM rather than directly assessing the learned dynamics. Even on ground-truth videos, IDM-based replay achieves only 87% success. Executability is also evaluated on just three tasks.

  • Limited qualitative results. It would be better to have comprehensive side-by-side comparisons of predicted and ground-truth video sequences.

Questions for authors
  1. Dexterous dynamics: How does the benchmark evaluate whether a world model accurately captures fine-grained contact dynamics, beyond generating the commanded finger motions?

  2. Simulation realism: How representative are the tasks and simplified renderings of real-world dexterous manipulation? How would the benchmark extend to compliant robot hands, deformable objects, and real-world visual observations?

  3. How should the resulting success rate be interpreted given the IDM's imperfect reconstruction and replay performance?

  4. Video prediction quality: Could the authors provide side-by-side ground-truth and generated video sequences across different models, ideally including both successful predictions and representative failures?

  5. Is there evidence that the proposed metrics correlate with downstream policy training, planning performance, or real world eval?

1 comment

We thank the reviewer for the constructive feedback and address the main concerns below.

  1. Novelty (W1). We agree that executability and interaction consistency are not unique to dexterous hands. The particular challenge of dexterous manipulation lies in coordinating multiple fingers, where similar object trajectories or task outcomes may conceal incorrect hand structures and local action responses. Building on existing evaluations, DexWorld-Bench extends the diagnostic granularity to the hand itself. It checks generated hand configurations through kinematic constraints, tests responses of the target body part through controlled action comparisons, and separately evaluates interaction outcomes and executability. Beyond single-finger control, the additional value lies in distinguishing hand-structure distortions, responses from the wrong body part, and interaction prediction errors. Our contribution is a dedicated dataset and diagnostic protocol addressing this blind spot.
  2. Complex, fine-grained contact (W2, Q1). Our evaluation examines not only whether fingers or arms move as commanded, but also whether these actions produce the correct physical interaction outcomes. In the interaction consistency dimension, we evaluate whether world models capture differences in physical interaction outcomes resulting from subtle changes in actions. For example, using successful and failed clips from the same grasping task, we assess whether generated videos preserve distinct outcomes, such as failing to lift the object, dropping it after lifting, or maintaining a stable grasp. In the task executability dimension, we further use a fixed inverse dynamics model (IDM) to recover actions from generated videos and replay them open-loop in the simulator, testing whether the generated visual trajectories can be translated into control sequences that successfully complete the task. This provides complementary evidence at the execution level. In the revised version, we plan to add contact-focused evaluations of interactions between dexterous hands and objects.
  3. Simulation realism (W3, Q2). DexJoCo currently includes grasping, tool use, and bimanual coordination tasks, which have functional counterparts in real-world dexterous manipulation. In the revised version, we will add evaluations of generalization to unseen tasks, manipulation of deformable objects, and real-world robotic tasks. We will also introduce Isaac Sim for rendering to support more realistic visual scenes.
  4. Task-executability evaluation (W4, Q3, Q5). The executability evaluation aims to assess the downstream execution utility of generated videos. Following the paradigm of video generation, action extraction, and task execution exemplified by Large Video Planner[1] and Vidar[2], we introduce an IDM as an action decoder (action expert) to form a complete execution evaluation pipeline. This examines whether generated visual trajectories can be translated into control sequences that successfully complete the task. The resulting success rate therefore reflects the simulation execution performance of the video model combined with a fixed IDM under specified conditions, including errors from action recovery and open-loop replay.
  5. Video generation quality comparisons (W5, Q4). In the revised manuscript, we will provide more comprehensive comparisons between predicted and ground-truth video sequences.

[1] Feng, Yao, et al. "Vidar: Embodied video diffusion model for generalist manipulation." arXiv preprint arXiv:2507.12898 (2025).

[2] Chen, Boyuan, et al. "Large video planner enables generalizable robot control." arXiv preprint arXiv:2512.15840 (2025).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.