- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents GPT-Policy, a harness layer to provide relevant offline and online context for GPT-6 Astra to accomplish robot manipulation tasks. The real world experiments checks the method's performance with different context types (human video, robot video and optionally action, goal image, history, and human interaction), and ablates demonstration information, visual observation triangulation, and action chunking. The paper discusses the contribution of individual components. Finally the simulation experiments with RoboDojo compare GPT-Policy with pi0.5 and two other black box foundation model calling methods.
Critical strengths
- Timely topic. This paper covers a very timely topic: as better closed source foundation models that are strong at agentic tool use become available, how to utilize them for robotics becomes very relevant. The paper provides real world demos with clean code release attached in the supplementary materials on this matter.
Critical weaknesses
-
Limited novelty relative to prior work: Using black box foundation models via tool calling, relevant context retrieval, or robotic domain harness has been studied since 2024. While this paper integrates many techniques, I am not fully convinced that this work provides substantial research contributions. For example, MOKA [1] utilizes in-context learning to use black box VLM for manipulation, and COME-robot [2] puts a black box in a closed-loop robot harness. Concurrently, many other works explore the similar idea of wrapping a harness layer around foundation models for robotic tasks, presumably because powerful new frontier models reduce the need for complex orchestration wrappers. The authors can do a better job at differentiating their work against its peers and highlighting their contributions.
-
Incomplete experiments or comparison. Although the authors evaluate their method on 10 real world tasks, on 6 of the tasks each setting is only evaluated 3 times and the results do not establish a clear comparison (GPT-Policy with the proposed context is tested, but not GPT-Policy without the proposed context). Neither are the baselines complete: pi0.5 is not evaluated on 4 real world tasks, and other call-black-box-vlm-with-harness-style baselines are not evaluated in real world, without which it becomes difficult to differentiate GPT-Policy from the myriad of related works. Finally, comparing with only pi0.5 on the selected tasks brings reservations: current-state-only VLA with no history is naturally disadvantaged at tasks emphasizing temporal context or interaction history such as pointed fruit pickup or tic-tac-toe. The authors could reconsider task selection or choose more suitable baselines for the existing tasks.
[1] Kuan Fang and Fangchen Liu and Pieter Abbeel and Sergey Levine. MOKA: Open-World Robotic Manipulation through Mark-Based Visual Prompting. Robotics: Science and Systems (RSS). 2024.
[2] Peiyuan Zhi and Zhiyuan Zhang and Yu Zhao and Muzhi Han and Zeyu Zhang and Zhitian Li and Ziyuan Jiao and Baoxiong Jia and Siyuan Huang. Closed-Loop Open-Vocabulary Mobile Manipulation with GPT-4V. 2025 IEEE International Conference on Robotics and Automation (ICRA).
Questions for authors
-
How do the different provided context type choices affect robot performance in RoboDojo simulation and other real world tasks (in addition to the 2 task pairs each context type choice is evaluated on)?
-
How do other harness-over-vlm baselines perform in real world tasks?
-
How much performance (in sim or real) comes from GPT-6 Astra itself versus from the proposed harness around it (e.g. can the authors expand Figure 5 / Table 7 and investigate the proposed method’s performance with different black-box VLM models)?
-
GPT-Policy takes minutes or even tens of minutes to execute a relatively short motion sequence and consumes millions of tokens as reported in Table 1, 3, and 7, which is far from ideal for real world usages. Have the authors utilized methods for time or token efficiency?
Rebuttal Opening
Dear Reviewers,
We sincerely thank you for the time and care you put into reviewing our work and for your thoughtful, constructive feedback.
As Reviewer WcUp noted, recent advances in VLMs make this a timely moment to study their use in robot control. Your suggestions led us to test how well demonstrations transfer to new layouts and to compare the latest VLM backbones and different control harnesses on real robots. We also clarified what demonstration keyframes and action references each contribute to robot control and how other researchers can use GPT-Policy as an accessible baseline. We believe it can inspire further work in agentic robotics.
We have replied to each reviewer. The revised PDF includes the new results and figures.