- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a benchmark designed to evaluate dexterous manipulation policies on a wide range of tasks on long horizon bimanual manipulation. They include a multitude of tasks and demonstrations. On the dataset side, the primary delta over prior work (DexJoco) is the number of tasks provided and the framework for composing stage-level tasks into longer horizon tasks.
Critical strengths
- Enabling task composition provides a useful interface for future work to study how the interplay of different tasks affect policy performance.
Critical weaknesses
-
Despite using a dexterous hand, most of the tasks studied don't really require any dexterity and could be tackled with a parallel jaw gripper. There are a few tasks requiring tool use (aspirate liquid), but for the most part, the tasks are long-horizon composition of simple primitives. The introduction motivates dexterous hands with “regrasp, twist objects, and manipulate tools” but the benchmark does not test any of this. Given that the whole point of using a dexterous embodiment is to tackle tasks which cannot be completed with a parallel jaw gripper, this seems like a major oversight.
-
While the paper states bimanual manipulation multiple times, nearly all of the tasks are for just one of the left/right hands. There is only one task(!) in the appendix that uses both hands, making bimanual a significant overclaim. Similarly, they say that they "support four end-effector types" but all data is collected with the Sharpa Hand and is kinematically retargeted, without any downstream verification of whether the demonstrations are successful.
Questions for authors
- If any IL baseline is tested with the parallel-jaw gripper embodiment, would it be able to succeed? Ideally tested without retargeted data, but I understand that it may not be possible to collect fresh parallel-jaw data for a task. If tasks can actually be completed without dexterous hands, that weakens the usefulness of the benchmark
- What percentage of the demonstrations are successful after retargeting? Does the benchmark provide a meaningful way to study cross-embodiment learning?
We sincerely thank the reviewer for the careful reading and constructive feedback. Below, we clarify the role of dexterity in the current task suite and the scope of the bimanual and cross-embodiment evaluation.
W1 & Q1. Dexterity requirements and parallel-jaw baselines
We agree that some tasks in the current suite may also be solvable with a parallel-jaw gripper. However, the goal of DexArena is not to claim that every task is exclusively solvable by a dexterous hand, but to provide a benchmark for learning manipulation policies under a dexterous-hand embodiment and high DoF action space.
The current submission does not include an IL baseline evaluated with the parallel-jaw gripper. At the same time, we acknowledge that a direct comparison between dexterous-hand and parallel-jaw policies would be valuable for characterizing where dexterous embodiment provides additional capability. We also see value in expanding the task suite with more skills involving regrasping, in-hand rotation, and dexterous tool use.
W2 & Q2. Cross-embodiment and bimanual scope
We would like to clarify that “retargeting” in our teleoperation pipeline refers to mapping the operator’s hand motion to the robot hand during data collection. The released demonstrations themselves contain Sharpa Hand trajectories. Among the 4,112 accepted Sharpa episodes, 3,222 (78%) successfully reproduce the task under open-loop action replay, and the released 3,000 demonstrations are selected from these replay-successful episodes.
This replay evaluation validates the collected Sharpa trajectories, but does not constitute quantitative validation of transferring these demonstrations to the other supported embodiments. We currently do not report policy or demonstration-transfer results for the Wuji Hand, XHand, or gripper.
Similarly, OpenPackage is the only stage that requires simultaneous use of both hands, while most other stages are executed by a single arm within the dual-arm environment. We therefore view broader cross-embodiment evaluation and richer genuinely bimanual tasks as valuable directions for extending the benchmark.