acceptodds
Under review as a conference paper at ICLR 2027

Are frontier LLMs ready for out-of-the-box robot manipulation?

Abstract

Embodied AI systems are often organized into System 1 and System 2. System 1 is typically a pretrained policy that generates actions at high frequency, whereas System 2 is often instantiated as a vision-enabled language model for high-level planning. We ask whether a large language model (LLM) can act as the policy for robot manipulation without task-specific finetuning. We call this setting LLM as policy. We evaluate three LLMs on all 42 RoboDojo tasks and compare their scores with 40 public policies. Astra and GPT-5.5 use the official 50-episode-per-task protocol; DeepSeek-Flash uses 10 episodes per task. GPT-6 Astra achieves 22.48% average success rate and 28.97 Score over 2,100 trials, ranking above every public entry. Yet GPT-5.5 and DeepSeek-Flash reach only 0.88% and 1.92% average success rate with the same post-processing. We find that Astra exhibits a sharply polarized capability profile. It generalizes well to tasks that require semantic understanding but not high-precision control. In contrast, it performs poorly on tasks that require precision, dynamic control, or complex bimanual coordination. In-context experiments show no aggregate benefit from one-shot demonstrations, while selected interaction traces show within-episode corrections under perturbations. Overall, the evaluated LLMs vary substantially in manipulation performance. Astra stands out and provides initial evidence for the potential of a general-purpose manipulation model, although reliable precision and dynamic control remain limitations in the evaluated setting.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Foundation model, Evaluation, Manipulation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_itzxOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the question of whether or not frontier LLMs are ready for out-of-the-box manipulation. The paper studies multiple frontier models and evaluates them in the RoboDojo simulation environment. The paper shows that while certain models can do well in tasks requiring semantic understanding, they perform poorly on tasks requiring precise control.

Critical strengths

The paper asks an important question, given the recent work around using state-of-the-art LLMs for robotic control.

Critical weaknesses
  • The findings are mostly not that surprising: it is widely known that LLMs are quite capable in semantic understanding, but they lack the capabilities for precise control. This is not a very surprising finding.
  • The conclusions are made on top of insufficient evidence. The paper focuses on only the RoboDojo environment and then makes the conclusion that “Astra stands out and provides initial evidence for the potential of a general-purpose manipulation model.” Even the non-RoboDojo evaluation is limited to just GPT Astra on a few simulation environments. To summarize, the evaluation is very limited to make conclusions.
  • The paper lacks hardware results and the reasons are a bit hard to believe. Yes, there could be some unsafe actions, but I don’t think one can make conclusions about the use of the latest models unless you also evaluate on hardware. I find this paper hard to accept without such evaluation. Even a convincing quantitative evaluation of unsafe actions or qualitative figures showing the unsafe behaviors and how often such behaviors occur would be helpful.
  • The paper presentation is a bit unclear as well.
    • Table 1 is very hard to read. It is unclear how to interpret the different columns — what are “Gen.”, “Prec.”, etc?
Questions for authors
  • The evaluation is quite insufficient:
    • How do all of these models do in other datasets, such as LIBERO, LIBERO-plus, or Simpler-Env?
    • How do these models perform on real robot arms in hardware?
      • I’m very skeptical of the claim that Astra repeatedly issued unsafe actions and that this is the basis for not evaluating in hardware. There have been multiple works that run these models on hardware and I think it’s unusual to halt the entire real-robot experiments due to this.
    • Are there other metrics you could report? Like unsafe behaviors (e.g., collisions) or near-misses (some more continuous metric instead of SR)?
    • The perturbation study can be expanded far beyond what is presented. What about other layouts and tasks beyond general_pickup?
  • How is move_eef implemented? I don’t see this anywhere. What happens if you vary the choice of implementation for move_eef? Does the performance change in precise manipulation?
  • What is the “post-processing” that is repeatedly referenced throughout the work?
  • Can you address the paper presentation concerns listed under Weaknesses?
Reviewer_Ptg6OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper provides an evaluation study of GPT 6 Astra, DeepSeek-Flash and GPT-5.5 on the RoboDojo simulation benchmark for LLM-driven robot control. Authors evaluate and benchmark these models against existing robot foundation models to provide insights into successes and failures, highlighting improvements and challenges in precision control. They further conduct experiments with in-context learning and visual/action space perturbations to better understand the behavior of Astra as a policy.

Critical strengths
  • Authors thoroughly benchmark Astra on the RoboDojo benchmark which provides a useful baseline for the community. Evaluations with GPT 5.5 and DeepSeek-Flash also demonstrate how LLMs have improved over recent years, and how open-source baselines still struggle in robot control.
  • The conclusions from one-shot demonstrations are useful to understand how LLMs-as-controllers may not benefit clearly with naive example task demonstrations. Experiments with varied visual/action/coordinate perturbations test the robustness of Astra as a policy, which is not studied before.
Critical weaknesses
  • My biggest concern with this work is the lack of novelty and strong new insights generated. The authors evaluate Astra with the standard protocol framework for LLM-driven robot control and do so on an existing benchmark with no new tasks/setup conditions. This paper reads more like an exploration of Astra as a robot policy rather than a novel contribution to the community.
  • From a robotics perspective the paper does not show a detailed analysis into the behavior demonstrated by Astra: what kind of 3D motion does it produce, what are the different safety issues in both sim & real and any suggestions to mitigate this for safe deployment. A further breakdown and comparison with robot foundation model rollouts like Pi-0.5, Cosmos might be more useful in understanding the gaps between LLM-driven manipulation and finetuned action space policies.
  • Authors do not talk about latency and costs associated with running these experiments, how it varies across models, long horizon tasks, etc.
  • While real robot experiments from RoboDojo were not setup due to safety concerns, some simple pick-and-place trials with standard robot arm manipulators/humanoids and comparisons with other LLMs/finetuned policies might be necessary to correlate simulation findings with real world deployment.
Questions for authors

1.Can the authors expand more on the novelty of the work beyond an exploration study of Astra? What more can the community learn beyond the fact that Astra can now do simple pick-and-place tasks? 2. Refer to Weakness #2: Compared to a finetuned policy how much does the motion distribution from Astra differ? 3. Please specify the inference latency and costs associated with each experiment

Reviewer_3GTuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether a business-level LLM can directly choose robot motion targets without a learned robot policy. The authors evaluate three models on RoboDojo where Astra achieves the highest overall score in this comparison, but its performance is uneven: good on open-ended tasks while struggling with the precise tasks.

Critical strengths
  1. The evaluation covers a broad set of tasks. And the chosen RoboDojo is a representative benchmark for the robot learning community.
  2. The differences across task types are interesting. Astra’s strengths on semantic tasks and weaknesses on precise control suggest that general-purpose LLMs and trained robot policies have different strengths.
Critical weaknesses
  1. The LLMs and compared baselines uses different input information. The LLMs receive wiki-derived task descriptions and scoring criteria in addition to the official instructions. The public robot policies were not evaluated with equivalent information. This matters because Astra’s largest gains are on open-ended and semantic tasks, where additional task descriptions may help. In such a way, the comparison is kind of unfair.

  2. The non-learned components handle target validation, motion planning, interpolation, and coordination between the arms. These components do not violate the paper’s definition of “LLM as policy,” but they still perform important control work. To some extent, the LLM as the so-call planners, to call those provided tools to complete the tasks.

Questions for authors
  1. How does Astra perform with only the official task instructions, without the wiki descriptions and scoring criteria?

  2. What are the inference latency, effective decision rate, and average number of calls per episode? And the token cost?

  3. What are the main failure types for GPT-5.5 and DeepSeek-Flash? Are they mostly perception errors, invalid actions, early termination, or failed execution? Are the failure types similarly across different LLMs? Are these failure types correlate to some ability of the LLMs, like SWE bench or spatial benchmarks?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.