acceptodds
Under review as a conference paper at ICLR 2027

OmniPiano: Diverse Dexterous Piano-Playing Challenges for Standard, Robust, Safe, and Multi-Agent RL

Abstract

Dexterous robotic manipulation remains a major challenge in robotics, exemplified by piano playing, which requires coordinated control of fingers and provides a demanding testbed for reinforcement learning (RL). Although previous work established a benchmark for robotic piano playing, it is restricted to two hands and does not support unified evaluation. To address this gap, we introduce OmniPiano, a benchmark that provides one to up to five Shadow Hands and supports standard, robust, safe, and multi-agent RL within a shared piano-playing task family. OmniPiano is developed for scalable and diverse piano-playing tasks with configurable perturbations, explicit safety constraints, and decentralized cooperation settings. Particularly, to facilitate usability and extendability, OmniPiano adopts a highly modular design and provides comprehensive tasks to support RL study of task performance, robustness, safety, and cooperation in dexterous control. With at least 912 task settings and 36 baseline algorithms, OmniPiano further incorporates LLM-based agents to broaden the evaluation scope and reveal new insights. Extensive evaluations show that state-of-the-art RL algorithms and frontier LLM-based agents still struggle on challenging piano-playing tasks, even under standard RL settings, highlighting substantial room for improvement in dexterous control. The code, dataset, tutorial, and demo are available at https://omnipiano.site.

Primary area:
datasets and benchmarks
Keywords:
Dexterous Piano Playing, Multi-hand Control, Reinforcement Learning, Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_ACXjOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This work presents a benchmark for dexterous robotic manipulation using piano playing as benchmark task. The benchmark extends RoboPianist along three different axes to assess control methods under different criteria. The Robust RL setting evaluates methods robustness to perturbations applied to the environment. Safet RL introduces cost functions that an agent must minimize while simultaneously maximizing rewards. The Multi-Agent RL setting allows the introduction of up to 5 hands that multiple agents must coordinate in order to play a song.

The benchmark is applied to evaluate a suite of commonly used RL algorithms, such as PPO and SAC, as well as using LLMs to synthesize programs for playing different piano songs.

Critical strengths

This work provides a systematic and challenging benchmark for dexterous control. By varying the environments along settings that evaluate robustness, safety, and multi-agent coordination, it can provide a standardized and reproducible evaluation benchmark for stress testing RL algorithms.

Critical weaknesses

Piano playing offers a challenging dexterous manipulation task, but I am unsure about the utility of evaluating models on this very narrow domain. The challenges of this piano playing task seem orthogonal to the major challenges for dexterous manipulation in more open world domains. For example, for real robots performing tasks around a home, some of the major challenges include perception, long horizon planning, and complex object interactions. All of these challenges are absent in piano playing, so it is unclear if optimizing for piano playing performance will offer significant real-world utility.

The technical innovation of this work appears fairly marginal. It does not offer much insight into improvements to methods. For a strong benchmark paper, I would generally expect it to provide some new insights on the limitations of existing methods and some methodological innovation to improve over those existing baselines. However, the discussion of the shortcomings of existing methods mainly boil down to both RL and LL agents “struggle on OmniPiano”. I am not sure if setting up a simulation environment is a significant enough technical contribution to meet the bar for ICLR.

My other major concern about this work is related to the organization of the main text. Many crucial components are relegated to the appendix. For example, the related work is a crucial component of a paper that provides context for the field. Similarly the LLM setup should also be in the main text. The main text should be self-contained and provide readers with the relevant information necessary to understand this work. Putting important information in the appendix amounts to circumventing the page limits by requiring readers to browse the appendix for pertinent information.

Questions for authors

Can the authors provide some insights on key limitations of existing RL and LLM methods that the OmniPiano benchmark identifies, and thoughts on how existing methods can be improved to address these shortcomings?

5 comments

We appreciate the reviewer for valuable suggestions of our works. In what follows, we provide our response to the reviewer's comments.The additional-experiments PDF can be found in https://drive.google.com/file/d/1DhR5ej92CIvfnTW8yA1SkBupkAz24szC/view?usp=sharing.

Q1: It is unclear if optimizing for piano playing performance will offer significant real-world utility

A1 We appreciate the reviewer’s concern about the benchmark’s broader relevance and practical utility. We would like to clarify that OmniPiano is designed as a controlled and reproducible testbed for evaluating RL algorithms on high-dimensional tasks requiring precise and coordinated actions, rather than for improving piano-playing performance as an end application or developing full-stack multimodal dexterous manipulation systems. Its value lies in systematically revealing the capabilities and limitations of RL algorithms under these challenging settings, for which a MuJoCo-based simulation is well suited.

Q2: Insights on key limitations of existing RL and LLM

A2: We thank the reviewer for this insightful question. Beyond the findings discussed in the paper, we further derive the following insights.

Limitations and Insight of Existing RL Methods:

  • Limited Control Decomposition and Coordination Our experiments suggest that the difficulty of dexterous RL is determined not only by the overall action-space dimensionality, but also by how control responsibilities are distributed and how agents interact. In standard RL, increasing the number of hands does not consistently improve performance, while restricted hand assignments lead to faster convergence and higher final returns (Figures 6 and 10). In MARL, distributing the same four-hand task across four agents rather than two generally improves performance, whereas imbalanced task assignments and coupled action spaces hinder learning (Figures 9 and 23).

Limitations and Insight of Existing LLM-based Frameworks:

  • Unstable iterative optimization as Figure 11, LLM-based frameworks exhibit considerable variance during iterative optimization, and program revisions can sometimes cause substantial performance regressions. Our trajectory analysis of 36 LLM optimization runs covering 357 evaluated iterations identifies two failure modes.
    • Diagnosis-to-Correction Gap. OmniPiano requires precise coordination of position, movement, and key-press timing. LLMs can correctly identify execution errors but struggle to translate that diagnosis into effective action changes.
    • Local-to-Global Coordination Gap. Changing one hand’s movements can interfere with the other. For example, modifying only the higher-register hand improved its note execution but increased missed notes in the neighboring hand. The LLM did not adequately check contact and interference between the hands
  • Limited Execution Robustness Beyond optimization stability, we further evaluate the execution robustness of the resulting controllers (Figure 3 in the submitted PDF containing additional experiments). All controllers are first trained in the nominal environment without position perturbations. During evaluation, we perturb the initial hand positions along the Y and Z axes using three offset ranges. The results show that PPO, SAC, TQC, and CrossQ consistently achieve higher average F1 than all four LLM-based frameworks at every nonzero perturbation level.
methodnominalY±10 / Z±5 mmY±25 / Z±10 mmY±50 / Z±20 mm
GPT-6 Astra0.60540.50470.25190.1349
Claude-Opus-50.45200.39760.26610.1543
DeepSeek-V4.1-Flash0.17260.16010.15110.1193
Gemini-3.8-Flash0.16770.10410.07770.0567
PPO0.56130.53540.43300.2503
SAC0.62240.56940.38440.2084
TD30.00050.00430.00200.0017
TQC0.59950.52910.33560.1945
CrossQ0.65840.55760.30950.1720
  • High and Model-dependent Inference Cost We also assess computational scalability by measuring token consumption and training time across different hand configurations (Figure 4 in the submitted PDF containing additional experiments). While token consumption varies substantially across LLMs (7.15–10.15M for GPT-6 Astra vs. 16.94–32.33M for DeepSeek-V4.1-Flash

Q3: Thoughts on how existing methods can be improved to address these shortcomings

A3: Our findings motivate the following thoughts:

  • Dependency-aware control decomposition. Structure high-dimensional policies around interacting agents or action groups, preserving necessary coordination while limiting unnecessary global dependencies.
  • Failure-informed robust training. Use failure analysis to identify challenging perturbations and gaps in state coverage, then prioritize these conditions during training.
  • Execution-verified refinement. Correct error diagnosis does not guarantee an effective correction. Verify candidate revisions through execution, checking whether they fix the target error, damage previously correct behavior. Use these results to guide revision acceptance and subsequent refinement.
  • Selective compression of interaction history. Replace repeatedly processed raw histories with compact records of critical states, effective revisions, and diagnosed failures. Compression should preserve information needed for subsequent decisions while reducing redundant context and token overhead.

Q4: Putting important information in the appendix

A4: We appreciate the reviewer's concern regarding the organization and accessibility of important information. We acknowledge that the page limit inevitably requires careful decisions about how to allocate content between the main text and appendix. We agree that the main text should be self-contained and provide all essential information needed to understand the benchmark. Our intention is not to require readers to consult the appendix to understand the core contributions. We would appreciate it if the reviewer could clarify which specific materials in the appendix they consider essential for understanding the main paper, so that we can better address this concern.

Regarding the LLM setup, we agree that its basic working paradigm should be introduced in the main text. We will add a concise description in Section 3.1, clarifying that the LLMs generate Python codes that produce open-loop actuator trajectories, rather than directly outputting low-level robot actions.

Regarding the related work, our organization was inspired by several recent benchmark papers at top-tier conferences, including HA-VLN [Li, H., et al., 2024], MMSearch [Jiang, D., et al., 2025], Robust Gymnasium [Gu, S., et al., 2025], MultiTrust [Zhang, Y., et al., 2024], and ImageNet-OOD [Yang, W., et al., 2024], which provide their detailed related work discussions in the appendices or supplementary materials. This helps maintain a coherent narrative and gives sufficient space to explain the benchmark's core contributions without interrupting the presentation with extensive literature reviews. Our Introduction and Table 1 already establish the research context and position OmniPiano relative to existing benchmarks, while the appendix provides a more comprehensive discussion of related work.

[Li, H., et al., 2024] Li, H., Li, M., Cheng, Z.Q., Dong, Y., Zhou, Y., He, J.Y., Dai, Q., Mitamura, T. and Hauptmann, A.G., 2024. Human-Aware Vision-and-Language Navigation: Bridging Simulation to Reality with Dynamic Human Interactions. Advances in Neural Information Processing Systems (NeurIPS 2024), 37, Datasets and Benchmarks Track.

[Jiang, D., et al., 2025] Jiang, D., Zhang, R., Guo, Z., Wu, Y., Lei, J., Qiu, P., Lu, P., Chen, Z., Song, G., Gao, P., Liu, Y., Li, C. and Li, H., 2025. MMSearch: Unveiling the Potential of Large Models as Multi-modal Search Engines. International Conference on Learning Representations (ICLR 2025).

[Gu, S., et al., 2025] Gu, S., Shi, L., Wen, M., Jin, M., Mazumdar, E., Chi, Y., Wierman, A. and Spanos, C., 2025. Robust Gymnasium: A Unified Modular Benchmark for Robust Reinforcement Learning. International Conference on Learning Representations (ICLR 2025).

[Zhang, Y., et al., 2024] Zhang, Y., Huang, Y., Sun, Y., Liu, C., Zhao, Z., Fang, Z., Wang, Y., Chen, H., Yang, X., Wei, X., Su, H., Dong, Y. and Zhu, J., 2024. MultiTrust: A Comprehensive Benchmark Towards Trustworthy Multimodal Large Language Models. Advances in Neural Information Processing Systems (NeurIPS 2024), 37, Datasets and Benchmarks Track.

[Yang, W., et al., 2024] Yang, W., Zhang, B. and Russakovsky, O., 2024. ImageNet-OOD: Deciphering Modern Out-of-Distribution Detection Algorithms. International Conference on Learning Representations (ICLR 2024).

Reviewer_4N3MOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces OmniPiano, a reinforcement learning benchmark that extends RoboPianist to support up to five dexterous hands, covering standard, robust, safe, and multi-agent RL. The authors evaluate several existing RL algorithms and agentic framework under different hand configurations, environmental perturbations, safety constraints, and multi-agent settings, and obtain several interesting findings, such as 1) omnipiano is a challenging benchmark that both RL and LLM-based agents still struggle with, 2) multi-agent settings enable the algorithms to learn better than centralized settings.

Critical strengths
  1. OmniPiano follows the framework of Robopianist and extends it to more challenging scenarios for evaluating standard, robust, safe, and multi-agent RL, with a large collection of baseline implementations.

  2. The paper is easy to follow. The website and documentation are clear.

  3. The design of the framework is modular, thus people can easily benchmark and test different different hand embodiments, perturbations, and safety constraints.

Critical weaknesses
  1. My main concern is regarding the motivation of the benchmark. Playing the piano is an interesting task demonstrating the dexterity of the hands and is a good test bed for RL and llm-based methods. However, from the paper it is unclear why we need to further extend the Robopianist to multiple hands. Also playing the piano is a relative toy task without much disturbance in real world, thus I don't quite get why we need to stay robust and safety on this specific task.

  2. I truly appreciate the authors implementing lots of baselines and thoroughly study the performance on the benchmark. In Section III, the authors show several findings from the evaluation. However, several experimental findings appear relatively expected, such as the observation that increasing the number of hands makes learning more difficult, restricting the action space improves convergence, or introducing stronger perturbations reduces performance. I would expect more compelling insights into existing algorithmic limitations or reveal new challenges that cannot be adequately studied using previous benchmarks.

  3. A good simulation benchmark is the one that can reflect the real world better. I know this work primary focusing on the design of the benchmark in sim. However, it is important to demonstrate some policies in real world to show that the findings in the simulation reflects what will happen in real world.

Questions for authors
  1. Providing some motivation of why OmniPiano is important and why we want to study this problem and extend the RoboPianist and why this is a relevant and important benchmark

  2. Beyond benchmarking existing algorithms, it would be helpful to provide more insights into their limitations and potential directions for improvement. For example, what capabilities are currently missing, and how might improvement on OmniPiano translate to improvements in other robotic manipulation tasks?

  3. Could the authors consider deploying a learned policy or an LLM-based framework on a real robotic hand? In particular, one motivation for evaluating robustness is to develop policies that can better handle real-world uncertainties. Even a small-scale real-world experiment could help demonstrate the practical relevance of the proposed benchmark.

  4. What is the evaluation reward? (Consider put it in the main paper) Also whether it reflects the task performance?

  5. RL-based methods and LLM-based frameworks represent quite different approaches to solving these tasks. It would be interesting to hear the authors' insights into their respective strengths and limitations. For example, which approach performs better under different settings, and how do they compare in terms of computational cost, and scalability?

7 comments

We would like to express our gratitude to the reviewer for their insightful feedback. We provide our response to the questions below. The additional-experiments PDF can be seen in https://drive.google.com/file/d/1DhR5ej92CIvfnTW8yA1SkBupkAz24szC/view?usp=sharing.

Q1: Why we need to extend the Robopianist to multiple hands

A1 Thanks for the important question. The motivation is not simply to increase task difficulty, but to move beyond the bimanual setting and systematically investigate the scalability and coordination ability of RL. With only two hands, it is difficult to study how control complexity evolves with the number of hands, how control responsibilities should be distributed among multiple controllers, and how inter-hand coupling affects decentralized execution. And OT reward removes fixed human fingering annotations, enabling flexible finger-to-key assignments across hand configurations. Our additional experiment illustrates the value of these evaluation dimensions (Figure 1 in the submitted PDF containing additional experiments). PPO and TRPO [John Schulman et al., 2015] perform similarly with the two-hand annotation-based reward (0.349 vs. 0.352), but differ under OT (0.414 vs. 0.340). Holding OT reward fixed, increasing from four to five hands improves PPO’s F1 from 0.505 to 0.592, while TRPO remains nearly unchanged (0.492 to 0.493). These results reveal differences in learning without prescribed fingering and exploiting additional hands, motivating multi-hand scaling alongside unified evaluation across the four RL paradigms.

[John Schulman et al., 2015] Schulman, J., Levine, S., Abbeel, P., Jordan, M. & Moritz, P.. (2015). Trust Region Policy Optimization. Proceedings of the 32nd International Conference on Machine Learning, in Proceedings of Machine Learning Research 37:1889-1897 Available from https://proceedings.mlr.press/v37/schulman15.html.

Q2: Why we need to stay robust and safety on this specific task

A2 We thank the reviewer for this question. Our primary goal is not to optimize piano playing as an end application, but to provide challenging and meaningful tasks that can advance RL research.

For Robust RL, our perturbations are designed to reflect common sources of uncertainty encountered in RL policy learning. These perturbations allow researchers to systematically evaluate whether RL algorithms can maintain performance under different sources of uncertainty. Specifically, observation noise mimics sensor inaccuracies, action noise represents actuator imprecision caused by calibration errors or mechanical wear, and reward noise simulates imperfect or noisy feedback during policy learning.

For Safe RL, our constraints are designed to represent different types of safety requirements in constrained RL, allowing researchers to systematically evaluate how algorithms balance task performance with constraint satisfaction. These constraints encourage RL algorithms to achieve high task performance while respecting physical safety limitations, a fundamental requirement for real-world robotic deployment. Specifically, joint-range constraints restrict excessive joint movements to prevent joint overextension and reduce the risk of mechanical damage to the robot. Hand-collision constraints limit unintended physical contact between robotic hands to reduce the risk of collision-induced damage.

By incorporating perturbations and safety constraints, OmniPiano aims to facilitate the development of RL algorithms that can achieve robust performance and satisfying safety constraints in high-dimensional, coordination-intensive, and precision-demanding settings, with relevance to practical applications.

Q3: Could the authors consider deploying a learned policy or an LLM-based framework on a real robotic hand?

A3: We sincerely appreciate the reviewer's suggestion. We agree that deploying learned policies or LLM-based frameworks on real robotic hands would provide valuable insights into their practical applicability and real-world performance.

However, we would like to clarify that the primary goal of OmniPiano is to provide a unified benchmark for systematically evaluating RL algorithms across different high-dimensional, coordination-intensive settings, rather than to develop or demonstrate full-stack robotic manipulation systems or sim-to-real transfer capabilities. We focus on isolating and investigating how factors such as task dimensionality, environmental perturbations, safety constraints, and multi-agent coordination affect the performance of RL algorithms, which is implemented by MuJoCo. We recognize the value of real-world deployment as a direction for future research. But hardware-specific integration and real-time execution on physical robotic hands are beyond the scope of this RL benchmark.

Q4: What is the evaluation reward, whether it reflects the task performance

A4: The evaluation reward is defined in Eq. (1) on page 3 (Section 2.2), combining key-press accuracy, fingertip-to-key matching, and an energy regularization term. Across our experiments, reward and F1 exhibit consistent positive trends. This is demonstrated in Standard RL (Figs. 5 and 11), Safe RL (Figs. 15–17 and 18–20), Robust RL (Figs. 12 and 13), and Multi-Agent RL (Fig. 23). F1 is an established metric for evaluating note-execution accuracy in robotic piano playing, as adopted in RoboPianist [Zakka, K., et al., 2023] and a recent real-robot piano-playing study [Azadjou et al., 2026]. By balancing precision and recall, it directly measures how accurately and completely the robot reproduces the target notes. The positive correlation between reward and F1 suggests that our reward effectively reflects task performance.

[Zakka, K., et al., 2023] Kevin Zakka, Philipp Wu, Laura Smith, Nimrod Gileadi, Taylor Howell, Xue Bin Peng, Sumeet Singh, Yuval Tassa, Pete Florence, Andy Zeng, Pieter Abbeel Proceedings of The 7th Conference on Robot Learning, PMLR 229:2975-2994, 2023.

[Azadjou, H., et al., 2026] Azadjou, H., Marjaninejad, A. and Valero-Cuevas, F.J., 2026. Perception in action: a robotic system that can teach itself to melodiously play music by ear. Journal of the Royal Society Interface, 23(238), 20250909.

Q5: It would be interesting to hear the authors' insights into RL-based methods and LLM-based frameworks' respective strengths and limitations.

A5: We thank the reviewer for this insightful question. Our experiments reveal complementary strengths and limitations of RL-based methods and LLM-based frameworks, as detailed in the upload PDF (https://drive.google.com/file/d/1DhR5ej92CIvfnTW8yA1SkBupkAz24szC/view?usp=sharing).

Interaction Efficiency and Computational Cost. To further compare optimization efficiency, we examine both environment interactions and computational resources (Figure 2 in the submitted PDF containing additional experiments). Across the three pieces, LLM-based frameworks require only 1,449–7,200 environment steps, substantially fewer than the approximately 5 million steps used by RL methods. This demonstrates their high interaction efficiency. However, LLM optimization incurs substantial token consumption (2.189–12.671M tokens across the evaluated frameworks), while RL training takes 2.06–22.98 hours on an i9-14900KF host with an RTX 4090. These results highlight different resource requirements: LLM-based optimization relies on repeated model inference, whereas RL relies on extensive environment interactions and policy training.

Optimization Stability. Despite their interaction efficiency, LLM-based frameworks exhibit considerable variance during iterative optimization, and program revisions can sometimes cause substantial performance regressions. Our trajectory analysis of 36 LLM optimization runs covering 357 evaluated iterations identifies two failure modes.

  • Diagnosis-to-Correction Gap. OmniPiano requires precise coordination of position, movement, and key-press timing. LLMs can correctly identify execution errors but struggle to translate that diagnosis into effective action changes
  • Local-to-Global Coordination Gap. Changing one hand’s movements can interfere with the other. For example, modifying only the hand playing the higher notes improved its note execution but increased missed notes in the neighboring hand. The LLM did not adequately check contact and interference between the hands

Execution Robustness. Beyond optimization stability, we further evaluate the execution robustness of the resulting controllers (Figure 3 in the submitted PDF containing additional experiments). All controllers are first trained in the nominal environment without position perturbations. During evaluation, we perturb the initial hand positions along the Y and Z axes using three offset ranges. The results show that PPO, SAC, TQC, and CrossQ consistently achieve higher average F1 than all four LLM-based frameworks at every nonzero perturbation level. This suggests that the learned RL policies exhibit stronger robustness to initial hand-position variations under the evaluated conditions.

methodnominalY±10 / Z±5 mmY±25 / Z±10 mmY±50 / Z±20 mm
GPT-6 Astra0.60540.50470.25190.1349
Claude-Opus-50.45200.39760.26610.1543
DeepSeek-V4.1-Flash0.17260.16010.15110.1193
Gemini-3.8-Flash0.16770.10410.07770.0567
PPO0.56130.53540.43300.2503
SAC0.62240.56940.38440.2084
TD30.00050.00430.00200.0017
TQC0.59950.52910.33560.1945
CrossQ0.65840.55760.30950.1720

Performance under Different Settings and Scalability. Regarding performance under different settings and scalability, we have evaluated both RL-based methods and LLM-based frameworks across different musical pieces and hand configurations, ranging from one to five hands, as detailed in Section 3.1 and Figure 6. The results show that both approaches remain competitive across different configurations, with LLM-based agents achieving higher F1 on the challenging five-hand WinterWind task.

We also assess computational scalability by measuring token consumption and training time across different hand configurations (Figure 4 in the submitted PDF). While token consumption varies substantially across LLMs (7.15–10.15M for GPT-6 Astra vs. 16.94–32.33M for DeepSeek-V4.1-Flash), RL training time increases moderately (2.34–3.62h for PPO and 14.90–16.54h for SAC), revealing distinct scaling patterns across models and algorithms.

Comments on the paper

9 comments
AuthorsOpenReview ↗

General Purpose - New experiments to further address the reviewer concerns

We thank the reviewers for their careful reading of the paper and their insightful and valuable feedback. We provide new experimental results and discussions to answer some common questions raised by reviewers. And the additional-experiments PDF can be seen in https://drive.google.com/file/d/1DhR5ej92CIvfnTW8yA1SkBupkAz24szC/view?usp=sharing.

Q3: New experiments to further address the reviewer concerns

A3: We added the following experiments and analyses to address the reviewers’ concerns:

  • Reward and hand-count comparison (Fig. 1): Compare PPO and TRPO under annotation-based and OT rewards, scaling from two to five hands to examine algorithm discrimination and scalability.
  • RL–LLM efficiency and cost analysis (Figs. 2 and 4): Compare performance against environment interactions, and report token consumption and training time across pieces and hand configurations.
  • Initial-position generalization tests (Fig. 3): Evaluate frozen controllers under three hand-position perturbation levels, using 20 seeds per level across three pieces.
  • LLM trajectory analysis: Examine 36 optimization runs covering 357 evaluated iterations to identify ineffective corrections and local revisions that disrupt overall coordination.

Figure numbers refer to the additional-experiments PDF.

AuthorsOpenReview ↗

General Purpose - Our contributions

We thank the reviewers for their careful reading of the paper and their insightful and valuable feedback. We provide new experimental results and discussions to answer some common questions raised by reviewers.

Q2: Our contributions

A2: To address these gaps, we introduce OmniPiano, a unified benchmark for evaluating RL algorithms across multiple RL regimes in high-dimensional coordinated control. Building upon RoboPianist, we extend the original setting with a series of practical challenges and configurable tasks.

  • We adopt an OT-based reward formulation [Peyré, G. and Cuturi, M., 2019] that eliminates reliance on the predefined fingering annotations used in RoboPianist. This allows learning algorithms to explore flexible fingering assignments.
  • We also expand the original two-hand setting to support up to five hands, increasing control dimensionality and the demands on inter-hand coordination while introducing greater potential for interference.
  • Beyond these general task extensions, we introduce targeted challenges across four RL regimes, allowing researchers to systematically investigate how different factors affect RL learning in high-dimensional, coordination-intensive tasks.

While existing benchmarks such as HumanoidBench [Sferrazza, C., et al., 2024] and Bi-DexHands [Chen, Y., et al., 2022] provide testbeds for high-dimensional whole-body and bimanual coordination, respectively, they do not offer a unified and configurable framework to jointly investigate the effects of hand count, environmental perturbations, explicit safety constraints, and multi-agent task allocation on high-dimensional cooperative RL learning.

Beyond the findings discussed in the paper, further analysis of our benchmark results highlights the following insights:

  • Control Decomposition and Coordination: Learning performance also depends on how control responsibilities are distributed and how agents interact. Effective decomposition can improve learning, while imbalanced allocations and strong coupling can hinder it.
  • Optimization Reliability in Precise Control: RL generally shows steadier optimization, whereas LLMs can correctly diagnose errors yet struggle to produce reliable corrections in precise control tasks such as piano playing.
  • Limited Execution Robustness: Both approaches are sensitive to initial hand-position changes, but several RL methods maintain higher average F1 than LLM-generated controllers under the tested perturbations.
  • High and Model-dependent Inference Cost: LLM-based optimization uses relatively few environment interactions but consumes millions of tokens. Token usage varies substantially across models and hand configurations.

[Sferrazza, C., et al., 2024] Sferrazza, C., Huang, D.M., Lin, X., Lee, Y. and Abbeel, P., 2024. HumanoidBench: Simulated humanoid benchmark for whole-body locomotion and manipulation. Proceedings of Robotics: Science and Systems (RSS 2024).

[Chen, Y., et al., 2022] Chen, Y., Wu, T., Wang, S., Feng, X., Jiang, J., McAleer, S., Dong, H., Lu, Z., Zhu, S.C. and Yang, Y., 2022. Towards human-level bimanual dexterous manipulation with reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS 2022), Datasets and Benchmarks Track, 35, pp. 5150–5163.

[Peyré, G. and Cuturi, M., 2019] Peyré, G. and Cuturi, M., 2019. Computational optimal transport. Foundations and Trends in Machine Learning, 11(5–6), pp. 355–607.

AuthorsOpenReview ↗

General Purpose - Why we study this

We thank the reviewers for their careful reading of the paper and their insightful and valuable feedback. We provide new experimental results and discussions to answer some common questions raised by reviewers.

Q1: Why we study this

A1: Learning policies for high-dimensional, coordination-intensive, and precision-demanding tasks is an important yet challenging problem in RL research [Chen, Y., et al., 2022] [Charlesworth, H. J., & Montana, G., 2021]. Our goal is to advance RL research by benchmarking four categories of RL algorithms in those tasks, rather than to benchmark dexterous manipulation systems. As also recognized by Reviewer 4N3M, piano playing provides a promising testbed, and RoboPianist [Zakka, K., et al., 2023] exemplifies this by combining a high-dimensional continuous action space with precise, temporally coordinated actions across multiple actuators. However, its original experimental setup is too limited to support a systematic investigation of this broader RL research problem. It primarily focuses on a fixed two-hand configuration with a dense key-press reward, without providing a unified and configurable framework to investigate how different factors, such as action dimension, environmental perturbations, safety constraints, and multi-agent task allocation, affect RL learning.

[Chen, Y., et al., 2022] Chen, Y., Wu, T., Wang, S., Feng, X., Jiang, J., Lu, Z., McAleer, S., Dong, H., Zhu, S.C. and Yang, Y., 2022. Towards human-level bimanual dexterous manipulation with reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS 2022), Datasets and Benchmarks Track, 35, pp. 5150–5163. Paper.

[Charlesworth, H. J., & Montana, G., 2021] Charlesworth, H.J. and Montana, G., 2021. Solving challenging dexterous manipulation tasks with trajectory optimisation and reinforcement learning. Proceedings of the 38th International Conference on Machine Learning (ICML 2021), PMLR 139, pp. 1496–1506. Paper.

[Zakka, K., et al., 2023] Zakka, K., Wu, P., Smith, L., Gileadi, N., Howell, T., Peng, X.B., Singh, S., Tassa, Y., Florence, P., Zeng, A. and Abbeel, P., 2023. RoboPianist: Dexterous piano playing with deep reinforcement learning. Proceedings of the 7th Conference on Robot Learning (CoRL 2023), PMLR 229, pp. 2975–2994.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.