- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces LabDex, a new benchmark for dexterous manipulation in a chemistry laboratory context. The benchmark's core features include a unified real-world and simulation environment for data collection as well as a task hierarchy with three levels of increasing complexity (atomic skills, compositional skills, long-horizon laboratory workflows). Multiple models are evaluated on the benchmark, including Action Chunking with Transformers (ACT), Diffusion Policy (DP) and the vision-language-action (VLA) model pi_0.5. The results show that pi_0.5 consistently achieves the highest performance in most atomic skills, though the long-horizon workflows remain an open challenge for the models.
Critical strengths
The paper's core strength is the novelty of the dataset itself with respect to commonly used dexterous manipulation tasks in laboratories. I am not familiar with existing works for grasping different laboratory equipment, though the related works and a review of existing literature show that existing works generally do not fully capture the task of grasping different assets in the laboratory. The practical implications of solving this problem are highly relevant to the community.
Similarly, the hierarchical task design makes sense in a laboratory setup. The real-world expert demonstrations and the augmentation via simulation provide a potentially valuable contribution by allowing robot policies to be trained and evaluated in a real laboratory environment. The selected assets and tasks span a wide range in the lab setup. The paper itself is also generally readable and well-structured.
Critical weaknesses
Despite the strengths of the core contribution, I believe the paper in its current form has several material weaknesses:
- The experiments are very limited in terms of supporting the core claims of the paper. The DP model scores an average of 0.02 across all atomic skill categories and achieves similar scores on the more complex tasks. In the appendix (Section A), the authors note that they "evaluate two additional dexterous-hand VLA models" (Kawaharazuka et al. (2025) and Li et al. (2025)) on "several Atomic Skills" (though the concrete skills are not specified) and that "neither model achieves meaningful task success under [their] evaluation setting." It is not clear from the paper why these models do not learn, why DP's success rate is so consistently low across all tasks.
- This becomes even more evident when examining related datasets such as the DexVerse (Yao et al. (2026)) paper cited in the work, where diffusion policy achieves success rates of over 90% on several pick-and-lift tasks. It is not necessarily important to show any high success rates at all, but understanding why the success rates are the way they are (e.g., what the different failure modes are and why multiple models fail to learn any tasks) is essential when this is the case. There is some limited discussion of this in the results, though it does not answer these questions.
- Somewhat related to the previous points, some experimental setups and results are not meaningfully discussed in the work at all, making it difficult to replicate the results. For instance, it is not clear to me whether the results using object distractors and novel-object generalizations were using the same pi_0.5 model as the other results or whether a new pi_0.5 model was trained. Similarly, the "six grasping tasks" used for the object generalization study are not named anywhere to the best of my knowledge.
- It is not clear to me why the simulator collection improves the data or the benchmark. The only discussion provided in the paper is that the results show "an overall trend consistent with that observed in the real-world experiments," but there is no indication that the simulator data itself helps improve real-world task performance. Similarly, there is sometimes confusion regarding whether a model was trained on real-world data, simulator data or a mix of both. Based on the description that "all models are trained using 200 real-world trajectories per task," it seems that only real data was used, but then that also does not explain why the simulator is needed at all.
- Ablation studies would be a great way to illustrate the importance of these components (e.g., the simulator, hierarchy of tasks etc.) in the benchmark. However, the existing one do not really support any claims of the paper. Table 6 shows that increasing the number of demonstrations improves the performance of models. Table 7 shows that object distractions reduce the success rate of the pi_0.5 model and Table 8 shows the success rate of unseen objects. From what I can tell, none of these demonstrate the effectiveness of the benchmark or the data itself or any of the contributions.
- The work does not adequately describe the limitations of the benchmark and directions for future work, only that the models have room for improvement.
Minor comments (do not affect the overall score):
- The qualitative figures (e.g., Figures 4 and 5) should clearly distinguish between simulated and real-world visualizations.
- The training details are near-duplicates in Section 4.1 and appendix Section A, though they each contain information the other does not.
- The paper citations in many cases do not include parentheses and this slightly affects the readability of the corresponding sentences. This appears to be a possible LaTeX formatting issue.
Based on the material weaknesses, my initial recommendation is a weak reject, though I am open to revising my score and look forward to the discussion.
Questions for authors
- What was the exact success criterion for each of the tasks?
- Why is there a difference in the first task between the independent success rate and sequential success rate? For instance, in the compositional skill "Transfer Beaker Water" in Table 3, DP and and pi_0.5's independent success rates significantly drop across the very first task whereas ACT's success rate nearly doubles in the sequential case. If there is such a high variance across the same task and the same model, how are the results statistically significant?
- What are the most common failure modes observed for the different models? For instance, in Table 8, the success rate on the 200ml beaker is shown to be 88% and the success rate on the 100ml beaker is shown to be 96% despite the latter being an object that was not seen during training. This does not make sense without additional information on the failure modes and why a trained policy generalizes better to a similar unseen object vs. the object it was trained on.
- Table 11 only shows 6 of the 26 atomic skills in simulation. Based on my understanding, Table 9 only refers to the results in the real-world setting. What are the results of the models on the other atomic skills? Are only the atomic skills simulated or also longer-horizon tasks?
- Why are different action chunk sizes used for DP, ACT and pi_0.5 (48, 48 and 50 respectively)? The appendix sheds some light on the different number of training steps for the different models, specifically to "keep the effective number of training epochs approximately consistent across the three task levels for each model", but I could not find any discussion of the action chunk size difference.