acceptodds
Under review as a conference paper at ICLR 2027

Make Code as Policy Great Again: Frontier Agents Write, Call, and Evolve Robot Tools

11 views

Abstract

Frontier models can control robots, but reasoning through every reach, grasp, and retreat makes manipulation slow and token-intensive. We revisit code as policy with a different division of labor: models build executable tools, code handles multi-phase motions, and models decide what to do next. We introduce URAI (Universal Robot–Agent Interface), which couples a programming agent that constructs robot tools with an execution agent that uses them in a feedback loop. The programming agent writes reusable and task-specific tools from task intent and refines them through execution feedback and human guidance. The execution agent selects and parameterizes these tools from current observations; each call runs a complete motion locally before returning control to the agent. Unlike delegating subsequent decisions to a generated program, this design retains model-level decision-making between tool executions. Validated tool revisions persist across episodes without updating foundation-model weights, and a shared GUI and API make the same tools available to humans and agents. Across five RoboDojo tasks and four frozen execution agents, URAI raises aggregate success from 18.0% to 53.0% relative to direct fingertip control, with the largest gain on Swap Blocks; with the same tools, a program written in advance reaches only 24% against 56% for two agents deciding after each call. All four agents also finish episodes faster, three of them 1.3–1.5 faster with 1.5–1.7 fewer execution-agent output tokens. We further evaluate URAI on seven real-world AgileX dual-arm tasks, spanning object manipulation, cloth folding, and human-interactive tic-tac-toe. URAI connects the coding and decision-making capabilities of frontier agents, organizing robot control around reusable tools that agents can both invoke and revise.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Agent, Robotics

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_bWXoOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces URAI (Universal Robot-Agent Interface), a framework for robotic manipulation that integrates a programming agent and an execution agent through one feedback loop. The programming agent is responsible for creating and refining robotic tools, while the execution agent selects and parameterizes these tools based on visual observations and execution feedback. Each tool executes a complete motion locally and returns control to the agent upon completion. This design bridges open-loop motion execution and model-driven feedback while enabling continuous tool refinement. The improvements in success rates, execution time, and token efficiency on RoboDojo simulation tasks and real-world manipulation tasks demonstrate the effectiveness of this method.

Critical strengths
  1. This paper proposes a framework that places greater emphasis on online agent feedback and decision-making compared to traditional Code-as-Policies approaches. By reintroducing the LLM for observation and decision-making after each complete tool execution, the proposed framework improves task execution flexibility and error recovery while retaining the efficiency of pre-programmed tool execution. Experimental results demonstrate improvements in task success rate, execution speed, and token efficiency.
  2. The proposed framework is evaluated on five RoboDojo simulation tasks and seven real-world robotic tasks, with four different foundation models used to validate its effectiveness. In particular, the authors conduct an ablation study using pre-generated programs built upon the same tool library, further demonstrating the benefits of integrating online agent decision-making between tool executions.
Critical weaknesses
  1. The core idea of this paper is to introduce LLM-based feedback and rearrangement between executions of pre-programmed tools. However, existing approaches such as CaP-X already support multi-turn interaction, execution feedback, and skill generation. URAI separates tool programming from online decision-making. While this design is reasonable from an engineering perspective, its fundamental novelty over prior methods remains unclear. Furthermore, the paper lacks direct experimental comparisons with related frameworks such as CaP-X.
  2. The paper emphasizes that the Programming Agent can generate, modify, and reuse robotic manipulation tools, while reducing the number of Execution Agent calls to improve execution efficiency. However, the reported execution time and token consumption do not account for the costs associated with tool generation, debugging, and validation. Moreover, existing frameworks such as CaP-X also support the reuse of previously generated skills. Although the authors provide a prewritten-program baseline using the same tools, a more comprehensive comparison with feedback-driven Code-as-Policy frameworks such as CaP-X would help clarify the advantages of URAI.
Questions for authors
  1. In the experiments, each task is evaluated with only five trials in RoboDojo and at most ten trials in real-world settings for each model and experimental condition. Although the authors report significant improvements, the limited number of trials makes it difficult to assess the reliability of the results for every task. Could the authors provide additional evaluations with more trials? Furthermore, the program baseline is only evaluated with GPT-6 Astra and Claude Opus 5.5. Why is this baseline not included for Claude Fable 5.1 and DeepSeek-V4-Flash?
  2. The paper mentions using RGB-D observations and depth information for 3D object localization, but provides limited details about the underlying visual perception pipeline. Does URAI rely entirely on the visual understanding capabilities of the foundation models, or does it incorporate additional perception tools, such as Segment Anything (SAM), for object segmentation and localization? If additional perception modules are used, are the same perception capabilities also provided to the Direct Control and Prewritten Program baselines to ensure a fair experimental comparison?
  3. One of the key advantages of the proposed framework is that the execution agent can make new decisions based on updated observations and execution feedback after each complete tool call, rather than relying entirely on a pre-generated task program. However, the current experiments do not systematically evaluate the framework's ability to adapt to unexpected changes in the environment during task execution. For example, could the authors consider introducing disturbances by manually changing the positions or states of target objects during execution, and evaluate whether URAI can replan based on updated observations and successfully complete the tasks, while reporting the corresponding success rates? A further comparison with the Prewritten Program baseline using the same tool library would help more directly demonstrate the benefits of the Agent-in-the-loop mechanism.
Reviewer_cWZHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper provides code as a robot policy framework leveraging an execution agent to select tools from observations and feedback and a programming agent to write and validate task-specific tools. A shared URAI validates and executes each motion locally based on the tools selected by the execution agent for each episode before allowing the programming agent to revise the tools without weight updates. A case study extending from rigid-object PnP tool to cloth manipulation was conducted. Using four execution agents applied to five RoboDojo tasks and on AgileX dual-arm tasks, URAI succeeds vastly more often while using fewer output tokens than step-by-step fingertip control. Using real robots, URAI completes tic-tac-toe and block into bowl tasks with greater efficiency than GPT-Policy.

Critical strengths

I appreciate that the authors validated URAI on a comprehensive test suite consisting of five simulated and seven real-world robotic tasks, demonstrating greater success rates, execution-phase time, and token efficiency compared to static programs over the same tools. The dual-arm coordination results on real-world tasks were particularly compelling. The folding clothes task shows that besides rigid object manipulation, deformable object manipulation can also be completed successfully and efficiently with URAI. Multiple frontier models as agents were tested which adds to the robustness of the work. Code-as-policy inherently allows for greater interpretability into agent decision making which could lead to more effective human-steering.

Critical weaknesses

There could be additional prompt or memory-only revision controls run as well as more simulator and dual-arm setups to be tested. I am more concerned about seeing specific tool selection and execution traces for some of the simulated and real-world tasks. There is a lack of transparency with the specific tools being rewritten, how often tools are reused, and the new tools being implemented. If more details are provided here, I would feel more confident about raising my score.

Questions for authors

Questions:

  • Could the authors provide an explanation for why using URAI leads to substantially lower motion calls besides the separation of the cost of writing the policy from the cost of running it?
  • A versioning mechanism where recurrent failures are passed to the programming agent and interruptions are passed to the execution agent is discussed, but what if there is incomplete observation information present?
  • How could costs associated with human guidance of tool programming be estimated and compared to the amount of human interventions in baselines?
  • Could the authors provide an example agent tool selection/execution chain for a particular tasks?
  • It would be interesting to see a comparison of how often new tools are implemented, how often reusable tools are revised, and how often existing tools are reused per task compared to baselines.
  • It would be helpful to see the prompts the authors used to guide the programming agent for the folding clothes task.
Reviewer_BiVuOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Strength

  • This paper addresses a good bottleneck in FM-based robot control -- improving token efficiency.
  • Simulation results are strong.
  • Real robots across different embodiments are also impressive.

Weaknesses

  • If the authors can compare with more closest related works in simulation it will be stronger.
  • The tool-evolution claim lacks a convincing quantitative evaluation.
  • The comparisons do not cleanly establish the claimed source or full cost of the gains.
Critical strengths

See summary.

Critical weaknesses

See summary.

Questions for authors

See summary.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.