- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper introduces a benchmark that attempts to study the language grounding properties of the VLA policies, with protocol designed to distinguish specific failure modes. The benchmark idea is timely and needed, as properly understanding how our wording of the tasks sent to the policies affects their performance has direct implications for the downstream applications of modern large robotic policies. However, the paper has critical issues in execution and validation of the studies.
Critical strengths
None
Critical weaknesses
The paper has following critical weaknesses, which make me chose "Clear rejection" rating.
- The paper is very hard to read, and overall reads like a technical report or large supplementary material, rather than research paper. It took me multiple passes over Sections 3.2-3.5 to understand all the details, which are presented in very bloated and convoluted manner.
- The most critical issue of the paper is verification of the claims. The paper claims that pi05-DROID and MolmoBot-DROID policies fail to complete manipulation, yet the reason seem to be caused by the embodiment mismatch. While authors try to adapt the viewpoints to those featured in the DROID dataset, in Section 4.2 L414-417 it is stated that "but never completes a grasp and place" in regards to the -DROID policy. Such a result is expected, as DROID platform utilizes Robotiq 2F-85 gripper, rather than standard Franka Hand gripper used in this benchmark. Hence the bad results are with high probability caused by the embodiment mismatch, where wrist camera sees completely different scene than it is used to from the training.
- Applicability of this benchmark is very limited. The benchmark requires it's own training data (recorded in the sim), that are not matched to any established embodiments used in the training of large-scale robotic policies, such as DROID, where questions asked in this benchmark actually make sense. Without it, the questions and answers are confounded with the embodiment mismatch, and small-scale data regime of this framework.
- The paper features a limited number of policies, which goes in hand with previous point. Using DROID embodiment would unlock plethora of off-shelf methods to evaluate their capabilities on for free.
Questions for authors
- Can you please clarify what adaptations were used in order to bridge the embodiment gap between DROID and "plain" Franka robot with Franka Hand? Why is -DROID achieving good success rates in RoboLab and MolmoSpaces, is still second best policy on RoboArena, yet fails in this benchmark?
- Why was DROID platform not considered in this study, when checkpoints used are DROID-specific?