acceptodds
Withdrawn from ICLR 2027

Agent as Policy for Robotic Manipulation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We demonstrate that a general-purpose agent can directly drive a physical robot throughout task execution without any task-specific or environment-specific training. We introduce Agent as Policy (AGP), which places task planning and execution under the agent's control. Given a task and a robot interface, the agent interprets visual evidence, writes executable programs, issues motion commands, and revises its actions in response to physical outcomes. This brings the agent's reasoning and programming capabilities into continuous interaction with the physical world. We study AGP across multiple real-world manipulation tasks spanning precision manipulation, dynamic motions, and deformable objects. These include assembly from human videos, block construction from goal images, dice flipping, targeted throwing, and bimanual towel folding. AGP achieves success rates of at least 80% in seven of eight task configurations and significantly outperforms previous agentic robot systems. We further study efficiency through task experience accumulation and find that reusing saved procedures and programs shortens execution time across repeated trials. These findings support a path for general-purpose agents to act as robot policies, extending their autonomy to physical manipulation through runtime reasoning, programming, and interaction.

Primary area:
applications to robotics, autonomy, planning
Keywords:
robotic manipulation, general-purpose agents, embodied AI, code generation, task and motion planning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_JsD9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Agent as Policy (AGP), which uses a general-purpose coding agent directly as a policy for robotic manipulation tasks. The agent requests observations and actions using a command-line interface and receives proprioceptive and visual observations as well as action outcomes, writing programs during execution to process observations and plan actions, and adapting based on motion feedback and observations from failed attempts. To improve efficiency, the paper proposes experience refinement, wherein an agent is informed by recorded experience that it maintains over multiple rollouts of the same task, and experience transfer, wherein relevant experience recorded by a stronger agent performing a given task is used to inform a weaker model performing the same task.

The approach is validated on five different manipulation tasks testing capabilities in long-horizon reasoning, action diversity and object diversity. The paper also compares performance across agent models on an assembly task, compares AGP against two baseline methods on three tasks, tests experience refinement and transfer on two tasks and one task respectively, and provides a case study on failure recovery. The authors report substantially higher success rates than the baselines and observe that past experience can facilitate more efficient task completion and can be transferred across agent models.

Critical strengths
  • AGP shows that an off-the-shelf coding agent supplied with only a thin robot interface can complete difficult real-world manipulation tasks zero-shot, including video-guided assembly with sub-millimetre clearances and bimanual towel folding, and succeeds in 23/25 trials against 3/25 and 0/25 for the baselines (Table 3). This is a non-obvious capability result and the main reason the paper merits consideration despite its thin methodological contribution.
Critical weaknesses
  • AGP's per-rollout cost and latency limit its practicality for repeated deployment, and the paper's efficiency comparison with ASPIRE obscures this since Section 4.2.3 applies ASPIRE's full preparation cost to every successful run.

    • Each AGP rollout incurs the (significant) full inference cost (mean 7.5--24.1 USD and 17.7--50.8 minutes per rollout; Table 1). In contrast, the programs generated by ASPIRE replay with no further inference cost, amortizing the high initial cost over many rollouts. Indeed, from the results for two-pair assembly in Table 3 (ASPIRE's fixed USD 18.90 vs AGP's USD 4.47 per rollout), I would expect ASPIRE to become cheaper per success after about 11 rollouts (assuming ASPIRE maintains its observed 40% success rate). Given that robots would often be expected to complete the same tasks many times over, this seems like a major limitation regarding the scalability of AGP, and a more representative comparison would consider accumulated cost over many rollouts.
    • Experience refinement and transfer (Section 4.3) reduce the per-rollout cost but not its linear scaling. Refinement reports only time (not token cost), and the ring task's 53% reduction mainly reflects a failed insertion in the first cycle (Appendix D.2). Transfer lets the cheaper Terra model execute, but each Terra rollout still costs roughly USD 1.75 (5.56M tokens at the rate implied by Table 2) and takes 12.5 minutes against ASPIRE's 2.9; assuming Terra's observed 80% success rate, I would expect ASPIRE to become cheaper per success after about 22 rollouts.
    • The agent's decision loop operates on a timescale of minutes (e.g., the recovery steps in Figure 3a each span 1--4 minutes), and throwing relies on a task-specific runtime that executes pre-authored timed programs (Appendix A.1), so the paper's claim of "dynamic motions" is overstated.
    • None of this is discussed as a limitation; the Broader Impact section (Appendix G) instead proposes AGP for collecting VLA training trajectories, which these costs make hard to justify compared to teleoperation.
  • No ablation isolates the central claim that using the agent directly as the runtime policy is what drives performance, and several comparative and statistical claims go beyond what the evaluation supports. This is particularly problematic since the methodological contribution is thin: AGP consists of an off-the-shelf agent, a seven-command robot interface and task prompts (Section 3, Appendix A.1), with experience reuse amounting to saved notes and scripts (Section 4.3). While simplicity is not itself a weakness, the paper's value therefore rests on its empirical claims.

    • A key missing ablation is the same agent and interface generating a single program up-front without runtime feedback, in the style of Code as Policies (Liang et al., 2022), which needs no simulator. Without it, gains cannot be attributed to runtime agency rather than the model's ability to write perception code (29 of 37 retained programs; Appendix E), so the findings say little beyond the specific model snapshot.
    • Baseline failures are attributed to the sim-to-real gap without reporting the baselines' success in simulation. GaP's 0/25 success also raises concerns about how faithfully it was reproduced.
    • The evaluation is sparse relative to the claims drawn from it. The agent comparison uses one task (Table 2), extended to a second task for only two models (Appendix C), baselines are compared only on three tasks the authors describe as "relatively easy" (Section 4.2.3), refinement uses a single five-run sequence on each of two tasks, and transfer is tested on one task and one model pair, with no confidence intervals reported. Despite this, Section 4.3 states that the refinement and transfer results "prove" their effects, though transfer compares 1/5 with 4/5 successes (by applying a two-sided Fisher's exact test I arrived at p$\approx$0.21), with time and token reductions computed from one run against four (Appendix D.3).
    • Protocol details are missing or weak. Budgets are never stated, the trial-inclusion policy is undescribed despite documented supervisor interventions (Appendices B.2 and C), and some successes are self-reported or leniently judged (Appendices D.2, I.5).
    • Appendix F's error analysis is never referenced in the main text, including a case where the agent incorrectly reports an incomplete fold as complete (Appendix F.5); this is especially relevant because AGP itself decides when a task is finished (Section 3.3), so such verification errors would go undetected without a human assessor.
    • All quantitative experiments use one tabletop setup with trials starting from predefined scenes (Section 4.1.2), so the claim of zero-shot execution in "previously unseen environments" (Sections 2.1 and 4.2.3) is tested only by a single qualitative demonstration on a second platform (Appendix H).
Questions for authors
  • Amortized cost (clarification): How do AGP's and ASPIRE's cumulative cost and time compare as a function of the number of rollouts, including failed trials? The current comparison applies ASPIRE's full preparation cost to each success.
  • Baseline validity (clarification): What are GaP's and ASPIRE's success rates in simulation? This would clarify whether the baselines' failures really reflect the sim-to-real gap.
Reviewer_4kbsOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AGP (Agent as Policy), which directly employs a frozen parameter general purpose multimodal large model agent as the robot policy. Leveraging runtime visual perception, on the fly programming and physical feedback closed loop, it accomplishes diverse real world robotic manipulation tasks including assembly, block construction, throwing and bimanual deformable object folding in a zero shot manner, and execution efficiency can be improved by storing and reusing task experience.

Critical strengths
  1. This work is well‑timed, as GPT‑6 has demonstrated strong embodied control capabilities which serve as a powerful foundation for this paradigm.
  2. Paradigm innovation: it treats the large‑model agent per se as the robot policy, which is distinctly different from conventional approaches of skill orchestration and pre‑generated code.
Critical weaknesses
  1. Tasks are highly time‑consuming (taking up to tens of minutes). Large‑model inference accounts for a large proportion of runtime, accompanied by high token consumption and expensive API inference costs, which is far from meeting the requirements for real‑time robot control. How can this issue be addressed?
  2. It only works for simple manipulation tasks and suffers prominent failures when applied to dexterous‑hand‑related tasks.
  3. Please elaborate on its differences from the prior CAP method as well as its corresponding advantages.
Questions for authors

See above Weaknesses.

Reviewer_W5BsOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Agent as Policy (AGP), which explores the capability of a general agent on real-robot manipulation. AGP acts directly as the robot policy throughout execution, with a two-part system organizing agents to interpret the task goals, understand the observations, and generate actions through an Agent-Robot Bridge. The authors evaluate several manipulation tasks on real robots and achieve promising results (over 80% success rates in seven of eight task configurations). They also enhance the execution efficiency by reusing successful programs and correction experience. Based on the exploration, the authors contribute an agentic system in physical interaction. The method demonstrates a new paradigm of strong general-purpose models, tool calling, and online feedback jointly managing fairly complex robot manipulation.

Critical strengths
  1. The research demonstrates the promising capabilities of general-purpose agents to interact continuously with the physical world.
  2. The paper shows cases where the agents automatically recover from several failure cases, illustrating how visual feedback influences action generations.
  3. The experience-accumulation and cross-model experiments point to a practical way of reducing inference overhead through documented experience.
Critical weaknesses
  1. The overall agentic system is a simple tool-calling and execution pipeline. The main contribution is more like a system-level combination between general base models and carefully designed interfaces, rather than a transferable algorithmic insight.
  2. The analysis and ablation studies on the system are limited. The paper does not sufficiently separate the contribution of the system organization from those of improved base-model capability, interface quality, task prompts, and execution protocols. The paper does not separate how much of the success comes from the AGP framework versus the capabilities of strong agents such as GPT-6 Astra.
  3. The evaluation protocols are weak. The number of tasks is limited, with each configurations 5–10 trials. The experiment provides insufficient support for robustness across different object instances, layouts, lighting, camera placements, or disturbances.
  4. The baseline comparison may introduce potential biases. GaP and ASPIRE are built and optimized in simulation and then transferred to the real robot, while AGP directly deployed in real-robot environments. The performance gap may be from the sim-to-real gap, interfaces, and preparation procedures, rather than the contributions of AGP's runtime architecture.
  5. Practical deployment efficiency remains a bottleneck. For example, some assembly and towel-folding tasks take tens of minutes on average, and inference costs are not low. The efficiency is unbearable in most real-world scenarios without further optimization.
Questions for authors
  1. What distinguishes AGP from existing closed-loop agent pipelines, such as Code as Policies with the close-looped execution feedback, or ReAct-style tool-calling agents? What insights in AGP are transferable and useful as base models and robot platforms change?
  2. What exactly do the "existing task prompts" and "practical guidance from previous runs" given to the preparation agent contain? Do they include human-written, task-specific strategies? Does the prior knowledge influence the evaluation result?
  3. How are the evaluation results if object positions, instances, lighting, camera viewpoints, and scene disturbances change?
  4. Can AGP be compared against baselines that also get online observation and code correction on the real robot, under the identical observations, agentic model and control interface?
  5. For tasks requiring continuous closed-loop control (e.g., force-controlled insertion, dynamic contact), how is AGP's inference latency handled?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.