- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This work presents a benchmark for dexterous robotic manipulation using piano playing as benchmark task. The benchmark extends RoboPianist along three different axes to assess control methods under different criteria. The Robust RL setting evaluates methods robustness to perturbations applied to the environment. Safet RL introduces cost functions that an agent must minimize while simultaneously maximizing rewards. The Multi-Agent RL setting allows the introduction of up to 5 hands that multiple agents must coordinate in order to play a song.
The benchmark is applied to evaluate a suite of commonly used RL algorithms, such as PPO and SAC, as well as using LLMs to synthesize programs for playing different piano songs.
Critical strengths
This work provides a systematic and challenging benchmark for dexterous control. By varying the environments along settings that evaluate robustness, safety, and multi-agent coordination, it can provide a standardized and reproducible evaluation benchmark for stress testing RL algorithms.
Critical weaknesses
Piano playing offers a challenging dexterous manipulation task, but I am unsure about the utility of evaluating models on this very narrow domain. The challenges of this piano playing task seem orthogonal to the major challenges for dexterous manipulation in more open world domains. For example, for real robots performing tasks around a home, some of the major challenges include perception, long horizon planning, and complex object interactions. All of these challenges are absent in piano playing, so it is unclear if optimizing for piano playing performance will offer significant real-world utility.
The technical innovation of this work appears fairly marginal. It does not offer much insight into improvements to methods. For a strong benchmark paper, I would generally expect it to provide some new insights on the limitations of existing methods and some methodological innovation to improve over those existing baselines. However, the discussion of the shortcomings of existing methods mainly boil down to both RL and LL agents “struggle on OmniPiano”. I am not sure if setting up a simulation environment is a significant enough technical contribution to meet the bar for ICLR.
My other major concern about this work is related to the organization of the main text. Many crucial components are relegated to the appendix. For example, the related work is a crucial component of a paper that provides context for the field. Similarly the LLM setup should also be in the main text. The main text should be self-contained and provide readers with the relevant information necessary to understand this work. Putting important information in the appendix amounts to circumventing the page limits by requiring readers to browse the appendix for pertinent information.
Questions for authors
Can the authors provide some insights on key limitations of existing RL and LLM methods that the OmniPiano benchmark identifies, and thoughts on how existing methods can be improved to address these shortcomings?
5 comments
Q2: Insights on key limitations of existing RL and LLM
A2: We thank the reviewer for this insightful question. Beyond the findings discussed in the paper, we further derive the following insights.
Limitations and Insight of Existing RL Methods:
- Limited Control Decomposition and Coordination Our experiments suggest that the difficulty of dexterous RL is determined not only by the overall action-space dimensionality, but also by how control responsibilities are distributed and how agents interact. In standard RL, increasing the number of hands does not consistently improve performance, while restricted hand assignments lead to faster convergence and higher final returns (Figures 6 and 10). In MARL, distributing the same four-hand task across four agents rather than two generally improves performance, whereas imbalanced task assignments and coupled action spaces hinder learning (Figures 9 and 23).
Limitations and Insight of Existing LLM-based Frameworks:
- Unstable iterative optimization
as Figure 11, LLM-based frameworks exhibit considerable variance during iterative optimization, and program revisions can sometimes cause substantial performance regressions. Our trajectory analysis of 36 LLM optimization runs covering 357 evaluated iterations identifies two failure modes.
- Diagnosis-to-Correction Gap. OmniPiano requires precise coordination of position, movement, and key-press timing. LLMs can correctly identify execution errors but struggle to translate that diagnosis into effective action changes.
- Local-to-Global Coordination Gap. Changing one hand’s movements can interfere with the other. For example, modifying only the higher-register hand improved its note execution but increased missed notes in the neighboring hand. The LLM did not adequately check contact and interference between the hands
- Limited Execution Robustness Beyond optimization stability, we further evaluate the execution robustness of the resulting controllers (Figure 3 in the submitted PDF containing additional experiments). All controllers are first trained in the nominal environment without position perturbations. During evaluation, we perturb the initial hand positions along the Y and Z axes using three offset ranges. The results show that PPO, SAC, TQC, and CrossQ consistently achieve higher average F1 than all four LLM-based frameworks at every nonzero perturbation level.
| method | nominal | Y±10 / Z±5 mm | Y±25 / Z±10 mm | Y±50 / Z±20 mm |
|---|---|---|---|---|
| GPT-6 Astra | 0.6054 | 0.5047 | 0.2519 | 0.1349 |
| Claude-Opus-5 | 0.4520 | 0.3976 | 0.2661 | 0.1543 |
| DeepSeek-V4.1-Flash | 0.1726 | 0.1601 | 0.1511 | 0.1193 |
| Gemini-3.8-Flash | 0.1677 | 0.1041 | 0.0777 | 0.0567 |
| PPO | 0.5613 | 0.5354 | 0.4330 | 0.2503 |
| SAC | 0.6224 | 0.5694 | 0.3844 | 0.2084 |
| TD3 | 0.0005 | 0.0043 | 0.0020 | 0.0017 |
| TQC | 0.5995 | 0.5291 | 0.3356 | 0.1945 |
| CrossQ | 0.6584 | 0.5576 | 0.3095 | 0.1720 |
- High and Model-dependent Inference Cost We also assess computational scalability by measuring token consumption and training time across different hand configurations (Figure 4 in the submitted PDF containing additional experiments). While token consumption varies substantially across LLMs (7.15–10.15M for GPT-6 Astra vs. 16.94–32.33M for DeepSeek-V4.1-Flash
Q3: Thoughts on how existing methods can be improved to address these shortcomings
A3: Our findings motivate the following thoughts:
- Dependency-aware control decomposition. Structure high-dimensional policies around interacting agents or action groups, preserving necessary coordination while limiting unnecessary global dependencies.
- Failure-informed robust training. Use failure analysis to identify challenging perturbations and gaps in state coverage, then prioritize these conditions during training.
- Execution-verified refinement. Correct error diagnosis does not guarantee an effective correction. Verify candidate revisions through execution, checking whether they fix the target error, damage previously correct behavior. Use these results to guide revision acceptance and subsequent refinement.
- Selective compression of interaction history. Replace repeatedly processed raw histories with compact records of critical states, effective revisions, and diagnosed failures. Compression should preserve information needed for subsequent decisions while reducing redundant context and token overhead.
Q4: Putting important information in the appendix
A4: We appreciate the reviewer's concern regarding the organization and accessibility of important information. We acknowledge that the page limit inevitably requires careful decisions about how to allocate content between the main text and appendix. We agree that the main text should be self-contained and provide all essential information needed to understand the benchmark. Our intention is not to require readers to consult the appendix to understand the core contributions. We would appreciate it if the reviewer could clarify which specific materials in the appendix they consider essential for understanding the main paper, so that we can better address this concern.
Regarding the LLM setup, we agree that its basic working paradigm should be introduced in the main text. We will add a concise description in Section 3.1, clarifying that the LLMs generate Python codes that produce open-loop actuator trajectories, rather than directly outputting low-level robot actions.
Regarding the related work, our organization was inspired by several recent benchmark papers at top-tier conferences, including HA-VLN [Li, H., et al., 2024], MMSearch [Jiang, D., et al., 2025], Robust Gymnasium [Gu, S., et al., 2025], MultiTrust [Zhang, Y., et al., 2024], and ImageNet-OOD [Yang, W., et al., 2024], which provide their detailed related work discussions in the appendices or supplementary materials. This helps maintain a coherent narrative and gives sufficient space to explain the benchmark's core contributions without interrupting the presentation with extensive literature reviews. Our Introduction and Table 1 already establish the research context and position OmniPiano relative to existing benchmarks, while the appendix provides a more comprehensive discussion of related work.
[Li, H., et al., 2024] Li, H., Li, M., Cheng, Z.Q., Dong, Y., Zhou, Y., He, J.Y., Dai, Q., Mitamura, T. and Hauptmann, A.G., 2024. Human-Aware Vision-and-Language Navigation: Bridging Simulation to Reality with Dynamic Human Interactions. Advances in Neural Information Processing Systems (NeurIPS 2024), 37, Datasets and Benchmarks Track.
[Jiang, D., et al., 2025] Jiang, D., Zhang, R., Guo, Z., Wu, Y., Lei, J., Qiu, P., Lu, P., Chen, Z., Song, G., Gao, P., Liu, Y., Li, C. and Li, H., 2025. MMSearch: Unveiling the Potential of Large Models as Multi-modal Search Engines. International Conference on Learning Representations (ICLR 2025).
[Gu, S., et al., 2025] Gu, S., Shi, L., Wen, M., Jin, M., Mazumdar, E., Chi, Y., Wierman, A. and Spanos, C., 2025. Robust Gymnasium: A Unified Modular Benchmark for Robust Reinforcement Learning. International Conference on Learning Representations (ICLR 2025).
[Zhang, Y., et al., 2024] Zhang, Y., Huang, Y., Sun, Y., Liu, C., Zhao, Z., Fang, Z., Wang, Y., Chen, H., Yang, X., Wei, X., Su, H., Dong, Y. and Zhu, J., 2024. MultiTrust: A Comprehensive Benchmark Towards Trustworthy Multimodal Large Language Models. Advances in Neural Information Processing Systems (NeurIPS 2024), 37, Datasets and Benchmarks Track.
[Yang, W., et al., 2024] Yang, W., Zhang, B. and Russakovsky, O., 2024. ImageNet-OOD: Deciphering Modern Out-of-Distribution Detection Algorithms. International Conference on Learning Representations (ICLR 2024).
We appreciate the reviewer for valuable suggestions of our works. In what follows, we provide our response to the reviewer's comments.The additional-experiments PDF can be found in https://drive.google.com/file/d/1DhR5ej92CIvfnTW8yA1SkBupkAz24szC/view?usp=sharing.
A1 We appreciate the reviewer’s concern about the benchmark’s broader relevance and practical utility. We would like to clarify that OmniPiano is designed as a controlled and reproducible testbed for evaluating RL algorithms on high-dimensional tasks requiring precise and coordinated actions, rather than for improving piano-playing performance as an end application or developing full-stack multimodal dexterous manipulation systems. Its value lies in systematically revealing the capabilities and limitations of RL algorithms under these challenging settings, for which a MuJoCo-based simulation is well suited.