acceptodds
Under review as a conference paper at ICLR 2027

In-Context Robot Learning with VLM Agents

26 views

Abstract

Enabling robots to adapt to unfamiliar environments as readily as humans remains a moonshot goal of embodied AI. No finite collection of demonstrations can cover every task and situation a robot will encounter, making the ability to learn from context at deployment essential for generalization. Such in-context learning (ICL), however, remains largely beyond the reach of existing robotic policies. The broad agentic capabilities of commercial vision-language models (VLMs), such as GPT-6 Astra, raise a compelling question: can these models learn from demonstrations and interaction feedback, then translate that information into executable robot behavior without gradient updates or persistent changes to task-specific parameters? We introduce GPT-Policy, a general-agent framework for in-context robot learning. GPT-Policy equips a VLM agent with a context compiler that organizes task-relevant information and an execution harness that supports geometric grounding, action chunking, and closed-loop decision-making through execution feedback. We evaluate GPT-Policy on ten real-world tasks spanning five context types and six RoboDojo simulation tasks. Across four real-world demonstration tasks, mean success rate increases from 10% without demonstrations to 80% with human videos or robot demonstrations that include action references. Videos provide procedural guidance, while action references can clarify motion details. Targeted ablations suggest that calibrated localization and action chunking improve success and execution efficiency. On these six RoboDojo tasks, one-shot GPT-Policy achieves 83.3% mean success. These results suggest that general-purpose VLMs can use deployment-time context to adapt robot behavior, while reliable physical execution remains a challenge.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot Learning, In-Context Learning, Embodied AI, Vision-Language Models, Robot Manipulation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_WcUpOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents GPT-Policy, a harness layer to provide relevant offline and online context for GPT-6 Astra to accomplish robot manipulation tasks. The real world experiments checks the method's performance with different context types (human video, robot video and optionally action, goal image, history, and human interaction), and ablates demonstration information, visual observation triangulation, and action chunking. The paper discusses the contribution of individual components. Finally the simulation experiments with RoboDojo compare GPT-Policy with pi0.5 and two other black box foundation model calling methods.

Critical strengths
  1. Timely topic. This paper covers a very timely topic: as better closed source foundation models that are strong at agentic tool use become available, how to utilize them for robotics becomes very relevant. The paper provides real world demos with clean code release attached in the supplementary materials on this matter.
Critical weaknesses
  1. Limited novelty relative to prior work: Using black box foundation models via tool calling, relevant context retrieval, or robotic domain harness has been studied since 2024. While this paper integrates many techniques, I am not fully convinced that this work provides substantial research contributions. For example, MOKA [1] utilizes in-context learning to use black box VLM for manipulation, and COME-robot [2] puts a black box in a closed-loop robot harness. Concurrently, many other works explore the similar idea of wrapping a harness layer around foundation models for robotic tasks, presumably because powerful new frontier models reduce the need for complex orchestration wrappers. The authors can do a better job at differentiating their work against its peers and highlighting their contributions.

  2. Incomplete experiments or comparison. Although the authors evaluate their method on 10 real world tasks, on 6 of the tasks each setting is only evaluated 3 times and the results do not establish a clear comparison (GPT-Policy with the proposed context is tested, but not GPT-Policy without the proposed context). Neither are the baselines complete: pi0.5 is not evaluated on 4 real world tasks, and other call-black-box-vlm-with-harness-style baselines are not evaluated in real world, without which it becomes difficult to differentiate GPT-Policy from the myriad of related works. Finally, comparing with only pi0.5 on the selected tasks brings reservations: current-state-only VLA with no history is naturally disadvantaged at tasks emphasizing temporal context or interaction history such as pointed fruit pickup or tic-tac-toe. The authors could reconsider task selection or choose more suitable baselines for the existing tasks.

[1] Kuan Fang and Fangchen Liu and Pieter Abbeel and Sergey Levine. MOKA: Open-World Robotic Manipulation through Mark-Based Visual Prompting. Robotics: Science and Systems (RSS). 2024.

[2] Peiyuan Zhi and Zhiyuan Zhang and Yu Zhao and Muzhi Han and Zeyu Zhang and Zhitian Li and Ziyuan Jiao and Baoxiong Jia and Siyuan Huang. Closed-Loop Open-Vocabulary Mobile Manipulation with GPT-4V. 2025 IEEE International Conference on Robotics and Automation (ICRA).

Questions for authors
  1. How do the different provided context type choices affect robot performance in RoboDojo simulation and other real world tasks (in addition to the 2 task pairs each context type choice is evaluated on)?

  2. How do other harness-over-vlm baselines perform in real world tasks?

  3. How much performance (in sim or real) comes from GPT-6 Astra itself versus from the proposed harness around it (e.g. can the authors expand Figure 5 / Table 7 and investigate the proposed method’s performance with different black-box VLM models)?

  4. GPT-Policy takes minutes or even tens of minutes to execute a relatively short motion sequence and consumes millions of tokens as reported in Table 1, 3, and 7, which is far from ideal for real world usages. Have the authors utilized methods for time or token efficiency?

Reviewer_q9LcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces GPT-Policy, which enables a VLM to control robots using demonstrations and interaction feedback without parameter updates. Real-robot experiments examine how different forms of context improve task completion.

Critical strengths

The ability to use human video demonstrations without robot action labels is appealing. The real-robot results provide encouraging evidence that deployment-time context can improve task execution.

Critical weaknesses

The evaluation does not sufficiently distinguish transferable skill learning from reliance on demonstration-specific procedural or numerical motion references. Controlled generalization tests under systematic changes in object poses and geometry would strengthen the central claim.

Questions for authors

Can the authors evaluate a fixed demonstration on held-out object poses and geometries where its recorded motion targets are no longer directly applicable? Could they also compare visual demonstrations with equally informative textual instructions to clarify the contribution of visual context?

Reviewer_DNLGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors investigate how to use in-context learning and VLMs to solve robotic manipulation tasks. They define a harness to utilize human / robot, demonstrations, goal images, history, etc. as context, and have the VLM come up with the action to control the robot. The framework is evaluated in real robot and simulated robot manipulation tasks.

Critical strengths
  • they focus on a framework that leverages in context learning abilities of VLM agents to solve robotic tasks, which is different than previous code as policy works
  • the overall paper is well organized and presented
Critical weaknesses
  • the paper is a system building effort rather than any novel component. the proposed harness mechanisms, are just putting together data modalities and feeding that to the VLM.
  • this feels reminiscent of wrappers and prompt engineering efforts in LLMs in GPT2 and 3 era, which were largely subsumed as the LLMs themselves got better and internally implemented these mechanisms.
Questions for authors

I think if the authors could focus more on novel insights, or showing that this is a strong and accessible baseline, etc., would improve the intellectual contributions of this work to the research community.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Rebuttal Opening

Dear Reviewers,

We sincerely thank you for the time and care you put into reviewing our work and for your thoughtful, constructive feedback.

As Reviewer WcUp noted, recent advances in VLMs make this a timely moment to study their use in robot control. Your suggestions led us to test how well demonstrations transfer to new layouts and to compare the latest VLM backbones and different control harnesses on real robots. We also clarified what demonstration keyframes and action references each contribute to robot control and how other researchers can use GPT-Policy as an accessible baseline. We believe it can inspire further work in agentic robotics.

We have replied to each reviewer. The revised PDF includes the new results and figures.

AuthorsOpenReview ↗

A Brief Follow-Up, with Thanks

Dear reviewers,

Hi! I hope this message finds you well. As the discussion period is nearing its end, it would be great to hear whether our responses have addressed the concerns you raised. Your insights are invaluable to us.

Thank you again for your time and effort :) Have a good day!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.