- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies the question of whether or not frontier LLMs are ready for out-of-the-box manipulation. The paper studies multiple frontier models and evaluates them in the RoboDojo simulation environment. The paper shows that while certain models can do well in tasks requiring semantic understanding, they perform poorly on tasks requiring precise control.
Critical strengths
The paper asks an important question, given the recent work around using state-of-the-art LLMs for robotic control.
Critical weaknesses
- The findings are mostly not that surprising: it is widely known that LLMs are quite capable in semantic understanding, but they lack the capabilities for precise control. This is not a very surprising finding.
- The conclusions are made on top of insufficient evidence. The paper focuses on only the RoboDojo environment and then makes the conclusion that “Astra stands out and provides initial evidence for the potential of a general-purpose manipulation model.” Even the non-RoboDojo evaluation is limited to just GPT Astra on a few simulation environments. To summarize, the evaluation is very limited to make conclusions.
- The paper lacks hardware results and the reasons are a bit hard to believe. Yes, there could be some unsafe actions, but I don’t think one can make conclusions about the use of the latest models unless you also evaluate on hardware. I find this paper hard to accept without such evaluation. Even a convincing quantitative evaluation of unsafe actions or qualitative figures showing the unsafe behaviors and how often such behaviors occur would be helpful.
- The paper presentation is a bit unclear as well.
- Table 1 is very hard to read. It is unclear how to interpret the different columns — what are “Gen.”, “Prec.”, etc?
Questions for authors
- The evaluation is quite insufficient:
- How do all of these models do in other datasets, such as LIBERO, LIBERO-plus, or Simpler-Env?
- How do these models perform on real robot arms in hardware?
- I’m very skeptical of the claim that Astra repeatedly issued unsafe actions and that this is the basis for not evaluating in hardware. There have been multiple works that run these models on hardware and I think it’s unusual to halt the entire real-robot experiments due to this.
- Are there other metrics you could report? Like unsafe behaviors (e.g., collisions) or near-misses (some more continuous metric instead of SR)?
- The perturbation study can be expanded far beyond what is presented. What about other layouts and tasks beyond general_pickup?
- How is move_eef implemented? I don’t see this anywhere. What happens if you vary the choice of implementation for move_eef? Does the performance change in precise manipulation?
- What is the “post-processing” that is repeatedly referenced throughout the work?
- Can you address the paper presentation concerns listed under Weaknesses?