acceptodds
Under review as a conference paper at ICLR 2027

LabDex: A Hierarchical Benchmark for Dexterous Manipulation in Laboratories

Abstract

Autonomous laboratories hold great promise for accelerating scientific discovery. To achieve this vision, robots are supposed to dexterously manipulate diverse labware and instruments and execute long-horizon, state-dependent experimental procedures. Yet existing benchmarks do not jointly capture dexterous hand use, real-world laboratory interactions, and multi-stage experimental procedures, limiting systematic training and evaluation. To bridge this gap, we introduce **LabDex**, a real-world dataset and benchmark for dexterous manipulation in chemistry laboratories, organized around a hierarchical task taxonomy spanning atomic skills, compositional tasks, and long-horizon experiments. First, LabDex is cross-platform and, for the first time, unifies real-world and simulation platforms under a common framework, providing standardized task definitions, demonstrations, and evaluation protocols. Second, LabDex systematically organizes chemistry laboratory operations into three interconnected levels: *Atomic Skills*, which characterize fundamental dexterous manipulation capabilities; *Compositional Skills*; and *Long-Horizon Laboratory Workflows*. This hierarchical design not only supports the evaluation of end-task performance, but also enables the analysis of how fundamental dexterous skills compose and influence more complex laboratory operations. We conduct cross-level evaluations of representative robot learning methods in both real-world and simulation environments. The experimental results validate the effectiveness of the LabDex task design and demonstration data, and show that the benchmark supports the training and systematic evaluation of existing robotic policies across laboratory dexterous manipulation tasks at different levels, providing a foundation for further research and development of autonomous laboratory robots.

Primary area:
datasets and benchmarks
Keywords:
Embodied AI, Dataset and Benchmark, Dexterous Manipulation in Laboratories

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_GzcvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces LabDex, a new benchmark for dexterous manipulation in a chemistry laboratory context. The benchmark's core features include a unified real-world and simulation environment for data collection as well as a task hierarchy with three levels of increasing complexity (atomic skills, compositional skills, long-horizon laboratory workflows). Multiple models are evaluated on the benchmark, including Action Chunking with Transformers (ACT), Diffusion Policy (DP) and the vision-language-action (VLA) model pi_0.5. The results show that pi_0.5 consistently achieves the highest performance in most atomic skills, though the long-horizon workflows remain an open challenge for the models.

Critical strengths

The paper's core strength is the novelty of the dataset itself with respect to commonly used dexterous manipulation tasks in laboratories. I am not familiar with existing works for grasping different laboratory equipment, though the related works and a review of existing literature show that existing works generally do not fully capture the task of grasping different assets in the laboratory. The practical implications of solving this problem are highly relevant to the community.

Similarly, the hierarchical task design makes sense in a laboratory setup. The real-world expert demonstrations and the augmentation via simulation provide a potentially valuable contribution by allowing robot policies to be trained and evaluated in a real laboratory environment. The selected assets and tasks span a wide range in the lab setup. The paper itself is also generally readable and well-structured.

Critical weaknesses

Despite the strengths of the core contribution, I believe the paper in its current form has several material weaknesses:

  • The experiments are very limited in terms of supporting the core claims of the paper. The DP model scores an average of 0.02 across all atomic skill categories and achieves similar scores on the more complex tasks. In the appendix (Section A), the authors note that they "evaluate two additional dexterous-hand VLA models" (Kawaharazuka et al. (2025) and Li et al. (2025)) on "several Atomic Skills" (though the concrete skills are not specified) and that "neither model achieves meaningful task success under [their] evaluation setting." It is not clear from the paper why these models do not learn, why DP's success rate is so consistently low across all tasks.
  • This becomes even more evident when examining related datasets such as the DexVerse (Yao et al. (2026)) paper cited in the work, where diffusion policy achieves success rates of over 90% on several pick-and-lift tasks. It is not necessarily important to show any high success rates at all, but understanding why the success rates are the way they are (e.g., what the different failure modes are and why multiple models fail to learn any tasks) is essential when this is the case. There is some limited discussion of this in the results, though it does not answer these questions.
  • Somewhat related to the previous points, some experimental setups and results are not meaningfully discussed in the work at all, making it difficult to replicate the results. For instance, it is not clear to me whether the results using object distractors and novel-object generalizations were using the same pi_0.5 model as the other results or whether a new pi_0.5 model was trained. Similarly, the "six grasping tasks" used for the object generalization study are not named anywhere to the best of my knowledge.
  • It is not clear to me why the simulator collection improves the data or the benchmark. The only discussion provided in the paper is that the results show "an overall trend consistent with that observed in the real-world experiments," but there is no indication that the simulator data itself helps improve real-world task performance. Similarly, there is sometimes confusion regarding whether a model was trained on real-world data, simulator data or a mix of both. Based on the description that "all models are trained using 200 real-world trajectories per task," it seems that only real data was used, but then that also does not explain why the simulator is needed at all.
  • Ablation studies would be a great way to illustrate the importance of these components (e.g., the simulator, hierarchy of tasks etc.) in the benchmark. However, the existing one do not really support any claims of the paper. Table 6 shows that increasing the number of demonstrations improves the performance of models. Table 7 shows that object distractions reduce the success rate of the pi_0.5 model and Table 8 shows the success rate of unseen objects. From what I can tell, none of these demonstrate the effectiveness of the benchmark or the data itself or any of the contributions.
  • The work does not adequately describe the limitations of the benchmark and directions for future work, only that the models have room for improvement.

Minor comments (do not affect the overall score):

  • The qualitative figures (e.g., Figures 4 and 5) should clearly distinguish between simulated and real-world visualizations.
  • The training details are near-duplicates in Section 4.1 and appendix Section A, though they each contain information the other does not.
  • The paper citations in many cases do not include parentheses and this slightly affects the readability of the corresponding sentences. This appears to be a possible LaTeX formatting issue.

Based on the material weaknesses, my initial recommendation is a weak reject, though I am open to revising my score and look forward to the discussion.

Questions for authors
  • What was the exact success criterion for each of the tasks?
  • Why is there a difference in the first task between the independent success rate and sequential success rate? For instance, in the compositional skill "Transfer Beaker Water" in Table 3, DP and and pi_0.5's independent success rates significantly drop across the very first task whereas ACT's success rate nearly doubles in the sequential case. If there is such a high variance across the same task and the same model, how are the results statistically significant?
  • What are the most common failure modes observed for the different models? For instance, in Table 8, the success rate on the 200ml beaker is shown to be 88% and the success rate on the 100ml beaker is shown to be 96% despite the latter being an object that was not seen during training. This does not make sense without additional information on the failure modes and why a trained policy generalizes better to a similar unseen object vs. the object it was trained on.
  • Table 11 only shows 6 of the 26 atomic skills in simulation. Based on my understanding, Table 9 only refers to the results in the real-world setting. What are the results of the models on the other atomic skills? Are only the atomic skills simulated or also longer-horizon tasks?
  • Why are different action chunk sizes used for DP, ACT and pi_0.5 (48, 48 and 50 respectively)? The appendix sheds some light on the different number of training steps for the different models, specifically to "keep the effective number of training epochs approximately consistent across the three task levels for each model", but I could not find any discussion of the action chunk size difference.
Reviewer_DTj9OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a dexterous laboratory benchmark and a dataset for dexterous manipulation for laboratory tasks, called LabDex. The task includes three different levels of laboratory manipulation tasks: atomic, compositional, and long-horizon skills. The benchmark covers both simulation and real-world setups, which include 8k real-world trajectories with atomic skill segmentations and 3D assets. The paper evaluates ACT, Diffusion Policy, and Pi0.5 in various tasks, in addition to distractor, data scaling, and novel object evaluations.

I recommend rejection because the novelty is limited and the motivation for dexterity in lab tasks is unclear. I would raise my score if the authors can present concrete reasons, tasks, gripper comparisons, lab-specific metrics, and dexterity-specific analysis to demonstrate the necessity of dexterity in lab tasks.

Critical strengths
  1. The paper proposes a series of laboratory dexterous tasks, datasets, and a comprehensive system setup. The evaluations are extensive, including evaluations on various skills, distractors, and novel generalization.
  2. 8K real-world trajectory dataset with atomic-level segmentation built into each demonstration and the 3d assets would be valuable for the field if released.
Critical weaknesses
  1. The novelty is limited. The contribution is mainly an integration of existing components. The gap claimed in section 2.2 does not hold. LabUtopia already includes hierarchical tasks. AutoBio has tasks with different difficulty levels. RoboChemist[1] does real-world laboratory tasks. The real distinction of this paper is the combination of dexterous hands with real-world data. However, the evaluation (task SR, distractors, data scaling, novel objects) is generic, which is not specific to dexterous hands or lab tasks.

  2. The motivation is unclear. Compared with prior work, LabDex differentiates from laboratory benchmarks by adding a dexterous hand. However, it is unclear why dexterity is necessary in laboratory tasks. A comparison of parallel grippers and dexterous hands is needed.

[1] Zongzheng Zhang, Chenghao Yue, Haobo Xu, Minwen Liao, Xianglin Qi, Huan-ang Gao, Ziwei Wang, Hao Zhao. RoboChemist: Long-Horizon and Safety-Compliant Robotic Chemical Experimentation. Proceedings of The 9th Conference on Robot Learning, PMLR 305:3537-3568, 2025.

Questions for authors
  1. The DP performance is not normal. DP performs nearly all zeros in all tasks, which is a huge gap compared to ACT. The authors should check their implementation or use the original DP implementation.
  2. Why is ACT's Sequential SR 26% higher than Independent SR in the "Pick 200ml Beaker" task in Table 3, while pi05 mostly has higher Independent SR? Appendix A says the first atomic skill uses the same reset protocol in both settings, so these numbers should match within noise. Where does the gap come from, evaluation variance or different training data?
  3. Can the authors provide the reset randomization range for training and testing? Without it, it is hard to assess spatial generalization.
  4. Can the authors provide some insights into when dexterity will be helpful? Which tasks in LabDex require dexterity, i.e., cannot be done with a parallel gripper?
Reviewer_1YhKOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

LabDex provides real-world and simulation resources for dexterous laboratory manipulation, organized into 26 atomic skills, 10 compositional skills, and one long-horizon workflow. It evaluates DP, ACT, and π0.5 using per-skill success and sequential progress. In this crowded benchmark landscape, the distinctive contribution remains insufficiently established, while DP's near-total failure in the real-world evaluation leaves the completeness of baseline validation uncertain.

Critical strengths

The study is relatively comprehensive in its real-world and simulation infrastructure, data collection, hierarchical task definitions, and per-skill evaluation.

Critical weaknesses

DP's unusually poor performance may reflect configuration or embodiment-adaptation issues that require investigation. Evaluating only one long-horizon workflow also leaves generalization across workflows unestablished.

Questions for authors
  1. What diagnostic or methodological value does the hierarchy provide beyond grouping tasks by complexity? Controlled stage transitions, expert-prefix replay, or unseen skill compositions could clarify whether the annotations reveal capabilities that ordinary task-level evaluation would miss.
  2. Lower sequential success can reflect upstream failures and different starting states, rather than insufficient memory. Please report stage-reach counts and conditional success rates, and compare the same downstream policy from expert-generated versus policy-generated intermediate states.
  3. Why does DP average only 2% atomic success and complete no compositional steps in the real-world evaluation? Please provide convergence and adaptation diagnostics covering action normalization, hand-joint weighting, execution frequency, and action-chunk deployment. How are pretraining and training-budget differences accounted for when interpreting model comparisons?
  4. Please provide numerical initial-state distributions, task-specific tolerances, success-checking procedures, and criteria for spills or unintended contacts. How are trajectories segmented and split across levels to prevent fragments of the same original execution from crossing training and testing?
  5. The related-work discussion appears too narrow to establish the benchmark's distinctiveness. Please broaden comparisons with closely related laboratory-automation, dexterous-manipulation, and hierarchical-task benchmarks, explicitly contrasting embodiments, task coverage, data, evaluation protocols, and generalization settings. Which concrete evaluation gaps remain unaddressed by existing resources?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.