acceptodds
Under review as a conference paper at ICLR 2027

LIBERO-Agent: Evaluating General-Purpose Agents for Direct Embodied Manipulation

Abstract

General-purpose agents can plan, use tools, and revise their behavior from feedback, but it remains unclear whether these capabilities transfer from digital environments to embodied manipulation. To investigate this question, we introduce LIBERO-Agent, an agent-native benchmark for evaluating these agents in robot manipulation tasks. Rather than asking agents to submit task-level Python control programs or operate through high-level robot skills, LIBERO-Agent provides an interactive robotic environment where agents can select which observations to inspect, process them with their own tools, and issue native action commands. LIBERO-Agent integrates 200 tasks into a common interaction framework and provides a 30-task primary suite that separates perception, short-horizon execution, and long-horizon composition. Results reveal a pronounced reliability gap: while agents perform well on perception and easy short-horizon tasks, their performance degrades substantially on hard short-horizon and long-horizon tasks. Richer observations improve short-horizon manipulation, while demonstration benefits depend on the agent and format. Among these agents, GPT-6 Astra achieves the strongest overall performance. Further analysis shows its major advantage lies in mechanism interaction, especially when sustained physical contact is needed, while its remaining failures stem from cross-stage interference and geometric errors.

Primary area:
datasets and benchmarks
Keywords:
General-Purpose Agents, Embodied Manipulation, Agent Evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XaZvOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors introduce LIBERO-Agent, a new benchmark for testing general-purpose agents using simulation for robot manipulation. Agents make their own decisions about what they see and how to act. There are no predefined skills for each type of task. There are 200 tasks included in the framework from other sources, and 30 primary tasks were used to test seven models/harnesses. These primary tasks covered perception, short-horizon, and long-horizon aspects. In addition to testing the agents themselves, the authors also looked at different types of observations and demonstrations as well as sustained contact and cross-stage failures. Overall, the results showed that the agents were able to identify targets well but had difficulty with difficult manipulation and multi-stage tasks.

Critical strengths
  1. The single interface built in the paper provides a good environment for studying how general-purpose agents can apply reasoning to produce actions in a unified way.
  2. The authors demonstrate some of the difficulties associated with maintaining long-term contact and interfering across stages.
  3. The paper is well written and easy to follow.
Critical weaknesses
  1. The need for a new benchmark is not justified. There are many other existing works covering generalization (LIBERO-Pro [1], LIBERO-Plus [2], RoboLab [3]), long-horizon reasoning (VLABench [4], RoboVoLo [5]), as well as memory (RoboMemArena [6], MIKASA-Robo [7], RMBench [8]). Although the agent-native interface is useful, the paper does not clearly show the value of its new 30-tasks benchmark.
  2. The interface itself is also limited to simulators, assets, robots, etc. It lacks real practical value and does not show scalability.

[1] LIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models Beyond Memorization.

[2] LIBERO-Plus: A Progressive Robustness Benchmark for Visual-Language-Action Models.

[3] RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies.

[4] VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Tasks.

[5] A Physical Orchestrator for Open-Vocabulary Long-Horizon Manipulation

[6] RoboMemArena: A Comprehensive and Challenging Robotic Memory Benchmark

[7] Memory, Benchmark & Robots: A Benchmark for Solving Complex Tasks with Reinforcement Learning

[8] RMBench: Memory-Dependent Robotic Manipulation Benchmark with Insights into Policy Design

Questions for authors

Please see the weakness.

Reviewer_V3swOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LIBERO-Agent evaluates general-purpose agents through autonomous observation selection and native robot-control commands. It integrates 200 tasks and evaluates seven agents on a 30-task suite covering perception, short-horizon manipulation, and long-horizon composition. Additional experiments examine observation modalities, demonstrations, and stage-level execution failures.

Critical strengths
  1. The interface leaves observation processing and action composition to the agent, providing a useful setting for evaluating embodied execution with limited task-specific scaffolding.

  2. The observation ablations and stage-level analysis go beyond reporting rankings and provide useful evidence about execution bottlenecks.

Critical weaknesses
  1. The evaluation is too limited to establish broad capability claims. Only 30 of the 200 tasks are evaluated, each with one fixed initial scene and three repetitions. This measures repeatability in specific scenes rather than robustness across scene variations.

  2. The main scoring scheme is insufficiently justified. Requiring success in all three runs, taking minimum stage completion, and assigning half the overall score to five hard tasks can strongly influence the reported gaps. Sensitivity to these choices is not examined.

  3. The capability interpretation remains confounded by the evaluation setup. Agents use different harnesses, while the effective budget is wall-clock time. Without complementary controls, differences in control ability are difficult to separate from harness behavior and inference latency.

Questions for authors
  1. How were the 30 primary tasks selected, and do the main findings hold across randomized initial scenes?

  2. How do the rankings and performance gaps change under average success rates and alternative score weights?

  3. Do the observed gaps persist under matched interaction budgets, with harness and inference-time effects accounted for?

Reviewer_YbsROpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces LIBERO-Agent, a benchmark for evaluating LLM agents on robot manipulation. Built on LIBERO and related task assets, the benchmark constructs a unified interface between agents and the simulation environment. Instead of executing tasks through fixed high-level robot skills, the agents are evaluated through selecting the available observations, generating lower-level robot action sequences, and conducting execution based on interaction feedback. The authors evaluate several agents using a "stable" metric that requires success in all three rollouts, plus a weighted overall score. The experiments show that most agents are expert at recognizing targets, but falling short in physical execution, especially in long-horizon tasks. In addition, richer observations like depth information improve short-horizon performances. A further analysis shows that the advantage of agents correlates with a sustained-contact ratio (SCR), while the remaining failures appear due to cross-stage interference and geometric errors.

Critical strengths
  1. The evaluation protocol focuses on the growing abilities of agents. The observation, tool-use, and code-execution capabilities of general-purpose agents are growing, but existing benchmarks usually focus on fixed observation and action interfaces.
  2. The authors provides useful design in evaluating capabilities of agents. Perception tasks score the identification ability rather than task completion, exposing the gap between correct identification and reliable execution. The split of short- and long-horizon tasks helps analyzing failures.
  3. The work provides evaluation across a variety of models, as well as ablations over observation modalities. The results reveal how depth information and demonstrations affect different agents, and the failure-case analyses make the evaluation results more interpretable.
Critical weaknesses
  1. The evaluation task suites have no reference baselines for validation. The paper reports no results for a classical perception and planning pipeline, or a VLA policy trained on LIBERO. This makes it hard to interpret what a specific success rate score means or whether the task difficulty is reasonable.
  2. The correspondence between the simulated benchmark and real manipulation is not stated. Several factors in real-world states, such as sensor noise, models, and real robot control, may change agents' error patterns and relative rankings.
  3. The experiment results lack statistical reliability. The primary suite has only 30 tasks, with 3 rollouts for each task. These tasks use the same initial state and simulator seed, so they do not measure generalization to scene variation. In addition, the remaining 170 tasks of the 200-task pool are never evaluated, which weakens the contributions of a 200-task benchmark.
  4. The contributions of models and harnesses are not separated. The paper evaluates models under different harness settings, which may introduce biases based on prompts, tool-call interfaces, and context management. The performance does not necessarily refer to a ranking of base models' capability.
Questions for authors
  1. Can the author report results baselines other than agentic models on the primary suite? This will put an upper bound on what the task set allows and make the difficulty easier to read.
  2. Could the author provide the evaluation results with randomized initial object poses, layouts, or seeds? Does the variation lead to the change of success rates and rankings?
  3. How are the 30 primary tasks selected from the 200-task pool? A full evaluation set will reduce the risk that it favors certain model families.
  4. The models are evaluated under different harness settings. Can several of the models be evaluated under a single, fixed harness, so that ranking differences can be attributed to the models?
1 comment
Reviewer_YbsROpenReview ↗

Response to Authors' Rebuttal

Thank you for the detailed rebuttal and the additional experiments. The VLA baselines, the task selection protocol, and the randomized-layout results address most of my concerns. From my perspective, the controlled study shows large harness effects, so the rankings should also consider comparisons of agent systems rather than base models. Overall, I appreciate the authors' efforts, and the rebuttal has improved my assessment of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.