- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper designed a dedicated benchmark for evaluating video generation models (text-conditioned, action-conditioned, WAMs) on DexDoJo dataset, benchmarking video generation qualities of hand and arm details, object trajectories, and executabilities for these models on simulated dexterous manipulation context.
Critical strengths
- This paper did a good job in doing controlled and rigorous benchmarking experiments on various video generation models and provided sufficient experimental evidence with illustrative figures. This is helpful for reproducing similar patterns and isolating different video generation features from different models.
- The benchmark design is well verified with calibration efforts. Ground-truth replay tests, comparisons between MLLM and human judgments, and IDM validation tests provide useful checks on evaluator reliability. The benchmark also includes failure interactions which rewards faithful generation of failed trajectories.
Critical weaknesses
- Modest technical contribution. The paper only benchmarks several video generation models, simply coming up with some heuristic metrics for comparison, without proposing promising and constructive alternatives for improvement. This makes this work simply a blog post for cage fighting existing methods.
- Executability is bounded by IDM. In actual manipulation replay setting, the IDM is trained with a very small scale of DexDoJo trajectories, so essentially the IDM is overfitting on these trajectories, which makes it extremely vulnerable to OOD scenarios. Even so, the IDM achieves only 0.87 success on ground-truth videos. Therefore, the IDM will easily hallucinate when the generated videos are slightly off the distribution of clean simulator images, making the executability benchmarking essentially 'good-looking' benchmarking. Moreover, inferring 16 actions from two endpoint images is too ambiguous for contact-rich motion.
- Dynamic, fine-grained and contact-rich motion. The key challenges in dexterous manipulation are (1) the high-dynamics of high-DoF robot systems, where close loop reactivity is crucial, and (2) ubiquitous contact and contact switching. Unfortunately, this work addresses none of these: the motions are mostly quasi-static open-loop action chunk replay without high dynamics and closed-loop feedback, and the contacts are mostly power grasps, without delicate finger gait switching or in-hand reorientation, and this benchmark doesn't even benchmark contact. Therefore, the benchmark might be informative for video generation on this specific domain, but will not be interesting to people actually working on real-world dexterous hands. It's also doubtful that the nuanced contact can be recovered by visual only input, where vision itself is not sufficient anymore under limited resolutions and heavy occlusions.
- Limited data scale. The benchmark only contains 220 full task sequences, which is very small. This scale of data can be easily overfitted by video generation models so it cannot establish correlation reliably. Also the generalization coverage is narrow since there is only one simulator and one embodiment, and no actual held-out tasks. Separate held-out results and tests across new tasks, objects, appearances, or embodiments would better support broad conclusions about these models.
- No real world experiments. Similar to 3, this work simply benchmarks video generation in a simulated domain, without real robot experiments for validation.
Questions for authors
Could authors provide more details on evaluation settings and protocols, e.g., fine-tuning budgets, IDM training, checkpoint selection, and generation settings?
We thank the reviewer for the constructive feedback and address the main concerns below.
[1] Bjorck, Johan, et al. "Gr00t n1: An open foundation model for generalist humanoid robots." arXiv preprint arXiv:2503.14734 (2025).
[2] Jang, Joel, et al. "Dreamgen: Unlocking generalization in robot learning through video world models." arXiv preprint arXiv:2505.12705 (2025).