- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors introduce LIBERO-Agent, a new benchmark for testing general-purpose agents using simulation for robot manipulation. Agents make their own decisions about what they see and how to act. There are no predefined skills for each type of task. There are 200 tasks included in the framework from other sources, and 30 primary tasks were used to test seven models/harnesses. These primary tasks covered perception, short-horizon, and long-horizon aspects. In addition to testing the agents themselves, the authors also looked at different types of observations and demonstrations as well as sustained contact and cross-stage failures. Overall, the results showed that the agents were able to identify targets well but had difficulty with difficult manipulation and multi-stage tasks.
Critical strengths
- The single interface built in the paper provides a good environment for studying how general-purpose agents can apply reasoning to produce actions in a unified way.
- The authors demonstrate some of the difficulties associated with maintaining long-term contact and interfering across stages.
- The paper is well written and easy to follow.
Critical weaknesses
- The need for a new benchmark is not justified. There are many other existing works covering generalization (LIBERO-Pro [1], LIBERO-Plus [2], RoboLab [3]), long-horizon reasoning (VLABench [4], RoboVoLo [5]), as well as memory (RoboMemArena [6], MIKASA-Robo [7], RMBench [8]). Although the agent-native interface is useful, the paper does not clearly show the value of its new 30-tasks benchmark.
- The interface itself is also limited to simulators, assets, robots, etc. It lacks real practical value and does not show scalability.
[1] LIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models Beyond Memorization.
[2] LIBERO-Plus: A Progressive Robustness Benchmark for Visual-Language-Action Models.
[3] RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies.
[4] VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Tasks.
[5] A Physical Orchestrator for Open-Vocabulary Long-Horizon Manipulation
[6] RoboMemArena: A Comprehensive and Challenging Robotic Memory Benchmark
[7] Memory, Benchmark & Robots: A Benchmark for Solving Complex Tasks with Reinforcement Learning
[8] RMBench: Memory-Dependent Robotic Manipulation Benchmark with Insights into Policy Design
Questions for authors
Please see the weakness.
Response to Authors' Rebuttal
Thank you for the detailed rebuttal and the additional experiments. The VLA baselines, the task selection protocol, and the randomized-layout results address most of my concerns. From my perspective, the controlled study shows large harness effects, so the rankings should also consider comparisons of agent systems rather than base models. Overall, I appreciate the authors' efforts, and the rebuttal has improved my assessment of the paper.