- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces URAI (Universal Robot-Agent Interface), a framework for robotic manipulation that integrates a programming agent and an execution agent through one feedback loop. The programming agent is responsible for creating and refining robotic tools, while the execution agent selects and parameterizes these tools based on visual observations and execution feedback. Each tool executes a complete motion locally and returns control to the agent upon completion. This design bridges open-loop motion execution and model-driven feedback while enabling continuous tool refinement. The improvements in success rates, execution time, and token efficiency on RoboDojo simulation tasks and real-world manipulation tasks demonstrate the effectiveness of this method.
Critical strengths
- This paper proposes a framework that places greater emphasis on online agent feedback and decision-making compared to traditional Code-as-Policies approaches. By reintroducing the LLM for observation and decision-making after each complete tool execution, the proposed framework improves task execution flexibility and error recovery while retaining the efficiency of pre-programmed tool execution. Experimental results demonstrate improvements in task success rate, execution speed, and token efficiency.
- The proposed framework is evaluated on five RoboDojo simulation tasks and seven real-world robotic tasks, with four different foundation models used to validate its effectiveness. In particular, the authors conduct an ablation study using pre-generated programs built upon the same tool library, further demonstrating the benefits of integrating online agent decision-making between tool executions.
Critical weaknesses
- The core idea of this paper is to introduce LLM-based feedback and rearrangement between executions of pre-programmed tools. However, existing approaches such as CaP-X already support multi-turn interaction, execution feedback, and skill generation. URAI separates tool programming from online decision-making. While this design is reasonable from an engineering perspective, its fundamental novelty over prior methods remains unclear. Furthermore, the paper lacks direct experimental comparisons with related frameworks such as CaP-X.
- The paper emphasizes that the Programming Agent can generate, modify, and reuse robotic manipulation tools, while reducing the number of Execution Agent calls to improve execution efficiency. However, the reported execution time and token consumption do not account for the costs associated with tool generation, debugging, and validation. Moreover, existing frameworks such as CaP-X also support the reuse of previously generated skills. Although the authors provide a prewritten-program baseline using the same tools, a more comprehensive comparison with feedback-driven Code-as-Policy frameworks such as CaP-X would help clarify the advantages of URAI.
Questions for authors
- In the experiments, each task is evaluated with only five trials in RoboDojo and at most ten trials in real-world settings for each model and experimental condition. Although the authors report significant improvements, the limited number of trials makes it difficult to assess the reliability of the results for every task. Could the authors provide additional evaluations with more trials? Furthermore, the program baseline is only evaluated with GPT-6 Astra and Claude Opus 5.5. Why is this baseline not included for Claude Fable 5.1 and DeepSeek-V4-Flash?
- The paper mentions using RGB-D observations and depth information for 3D object localization, but provides limited details about the underlying visual perception pipeline. Does URAI rely entirely on the visual understanding capabilities of the foundation models, or does it incorporate additional perception tools, such as Segment Anything (SAM), for object segmentation and localization? If additional perception modules are used, are the same perception capabilities also provided to the Direct Control and Prewritten Program baselines to ensure a fair experimental comparison?
- One of the key advantages of the proposed framework is that the execution agent can make new decisions based on updated observations and execution feedback after each complete tool call, rather than relying entirely on a pre-generated task program. However, the current experiments do not systematically evaluate the framework's ability to adapt to unexpected changes in the environment during task execution. For example, could the authors consider introducing disturbances by manually changing the positions or states of target objects during execution, and evaluate whether URAI can replan based on updated observations and successfully complete the tasks, while reporting the corresponding success rates? A further comparison with the Prewritten Program baseline using the same tool library would help more directly demonstrate the benefits of the Agent-in-the-loop mechanism.