acceptodds
Under review as a conference paper at ICLR 2027

OpenRUA: Robot-Use Agents Are Zero-Shot Visuomotor Policies

Abstract

Coding agents are extending their reach into the physical world by writing and executing robot control programs. One might expect the agents to use the same mature software stack that engineers have developed over decades. Yet existing work surrounds these agents with specially engineered interfaces for robot use, wrapping learned policies as tools or supplying hand-crafted perception and control primitives. We introduce OpenRUA, a harness that bypasses bespoke abstraction layers by providing off-the-shelf coding agents with only terminal access to the robot's native software interface ROS 2. OpenRUA employs a zero-abstraction workspace-as-harness design, only offering ROS 2 documentation and basic tools while leaving the coding agent to organize its own work without orchestrating any agentic workflow. OpenRUA achieves success rates of 99.0% on CaP-Bench and 87.0% on LIBERO-PRO, demonstrating that an off-the-shelf coding agent can serve as a zero-shot visuomotor policy through the robot's native interface, without bespoke primitives or task-specific training. (1) For perception, the agent spontaneously writes programs that process raw sensory inputs and derive metric measurements in 96.80% of episodes. (2) For manipulation, the agent spontaneously builds motion-control clients in 95.87% of episodes and closed-loop control programs in 50.13% of episodes. Our code is available at https://anonymous.4open.science/r/OpenRUA-34210.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot Control, Robot-Use Agents, Coding Agents, Visuomotor Policies, Code as Policies, ROS 2

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_73qqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

OpenRUA gives off-the-shelf coding agents terminal access to native ROS 2 interfaces and a workspace containing machine documentation and starter tools, allowing the agents to acquire observations and write and execute robot-control programs without a prescribed workflow. The main evaluation reports 99.0% success on CaP-Bench, 87.0% on LIBERO-PRO, and 39.4% on RoboCasa365. The paper additionally analyzes agent-written perception and control programs, alternative model configurations and computational costs, and failed episodes.

Critical strengths
  • Extensive simulation evaluation. The paper evaluates tasks across CaP-Bench, LIBERO-PRO, and RoboCasa365, providing substantial empirical coverage of the system’s capabilities in simulation.
  • The extensive code listings and explanations in the appendix, in my view, reflect the authors’ confidence in the authenticity of their work.
Critical weaknesses
  • Unclear contribution beyond off-the-shelf agents. The baseline comparisons involve different underlying models: OpenRUA uses Opus 5, whereas ASPIRE uses Opus 4.6 and Harness VLA uses Opus 4.7 or GPT-5.5. These model differences make it difficult to attribute the reported performance gains to OpenRUA's harness rather than the underlying agents' capabilities. Without matched-model interface comparisons or workspace ablations, the harness's own contribution remains insufficiently established.
  • Insufficient analysis of the trade-off between execution time and success rate. Figure 6 and Appendix E analyze agent responses time and computational costs, but do not quantify the trade-off between actual task execution time and success. Moreover, simulation pauses during agent deliberation, excluding the effects of inference delays on a continuously evolving environment. It therefore remains unclear whether the reported success rates are achievable within practically useful time budgets.

Besides, it is also lack of real-world validation. All experiments on CaP-Bench, LIBERO-PRO, and RoboCasa365 are conducted in simulation, with no physical-robot validation. The generated programs can reliably and safely complete tasks on physical robots remains unverified. This substantially limits the evidence for the system's practical applicability.

Questions for authors

In limitation part, you note that agents sometimes implement their own force thresholds and stopping conditions. When autonomously constructing their control programs, do agents also spontaneously build a more systematic safety framework, such as motion limits or collision checks?

7 comments
AuthorsOpenReview ↗

Response to Reviewer 73qq (Part 1/5): Contribution & Matched-Model Comparison (A1a)

Thank you for recognizing our extensive benchmark evaluation. If we understood you correctly, your central concerns were two: 1) that our harness's contribution was unclear, and 2) that our evaluation lacked real-world validation. We have addressed both, and the result has strengthened our work. Specifically, in this rebuttal and the revised paper (new content in blue), we add matched-model comparisons and workspace ablations (A1; Section 4.6), alongside time-budget analyses, physical-robot experiments, and safety evaluations (A2–A4; Sections 4.7 and 5, Appendix K).

 


▶ W1: Unclear contribution beyond off-the-shelf agents. The baseline comparisons involve different underlying models: OpenRUA uses Opus 5, whereas ASPIRE uses Opus 4.6 and Harness VLA uses Opus 4.7 or GPT-5.5. These model differences make it difficult to attribute the reported performance gains to OpenRUA's harness rather than the underlying agents' capabilities. Without matched-model interface comparisons or workspace ablations, the harness's own contribution remains insufficiently established.

A1: Our contribution lies precisely in showing that, beyond off-the-shelf agents, robot manipulation does not have to rely on stacking complex modules or performance tricks on top of them for extra gains. Instead, following the spirit of the Bitter Lesson, we explicitly set out to disintermediate or minimize the harness layer between the agent and the robot via simple ROS 2 terminal access plus a thin workspace. Our aim is to prompt the community to rethink current trends toward complex, specialized harness engineering as models grow more capable. To support this position, we add experimental evidence below.

First, we compare OpenRUA with a strong baseline, ASPIRE, using its latest published results on the same Claude Code + Opus 5 (high reasoning effort) backbone across CaP-Bench and LIBERO-PRO (all 8 cells). To give a clear view of the additional harness engineering effort in both methods, we use harness code (lines) and harness doc/prompt (words) as rough proxies; here, code refers to the pre-built Python tools, custom APIs, and workflow scripts directly supplied around the agent, and doc/prompt refers to the guide docs, workflow prompts, and machine descriptions provided before task execution. Both methods are evaluated under each benchmark's official success predicates and paused-clock, sensor-only rules (no privileged object poses or test-time oracle access; initial-state protocols in Appendix B.1). In addition, OpenRUA runs zero-shot from a fresh workspace in every episode (100 trials/task on CaP-Bench; official initial states 0–9/task on LIBERO-PRO), whereas ASPIRE runs with task-specific skill experience accumulated beforehand in additional episodes.

MethodRobot interfaceEvaluation paradigmCaP-Bench ↑LIBERO-PRO ↑Harness code ↓Harness doc/prompt ↓
ASPIREStructured primitivesWith prior experience (skill library)98.86%94.45%5,391 lines24,565 words
OpenRUA (ours)Native ROS 2 + thin starter workspaceZero-shot (fresh workspace per episode)99.00%87.00%337 lines (16× less)3,528 words (7× less)

The results show that, under the same Opus 5 backbone, OpenRUA uses 16x less harness code (337 vs. 5,391 lines) and 7x less doc/prompt text (3,528 vs. 24,565 words) than ASPIRE, while matching it on CaP-Bench (99.00% vs. 98.86%) and reaching competitive zero-shot performance on LIBERO-PRO (87.00% vs. 94.45%). Here, ASPIRE's higher LIBERO-PRO score is consistent with its pre-evaluation experience. While OpenRUA can also carry experience across episodes in its workspace, we intentionally evaluate zero-shot to show that native ROS 2 access and a minimal workspace are already enough to unlock a strong model's potential without prior experience or heavy harness engineering.

AuthorsOpenReview ↗

Response to Reviewer 73qq (Part 2/5): Workspace Ablation Across Models (A1b)

Second, we conduct a workspace ablation on the LIBERO-PRO Goal-Position suite (10 tasks, official initial states 0–9; 100 episodes per setting). Specifically, we retain the core workspace design, i.e., giving the agent native ROS 2 terminal access in a project directory, while removing all starter documentation and tools and leaving only the machine.yaml configuration. We evaluate this setting across both Opus 5 and the stronger Opus 5.5 under the same Claude Code (high reasoning effort) setup and identical evaluation rules.

MethodModelEvaluation paradigmSuccess ↑Mean time* ↓Harness code ↓Harness doc/prompt ↓
ASPIREOpus 5With prior experience97.0%–5,391 lines24,565 words
OpenRUAOpus 5Zero-shot91.0%55.8 min337 lines3,528 words
OpenRUA w/o docs/toolsOpus 5Zero-shot88.0%59.3 min0 lines856 words
OpenRUA w/o docs/toolsOpus 5.5Zero-shot91.0%21.5 min0 lines856 words

*Note: Mean time is the end-to-end active wall-clock time per episode, including model reasoning, code execution, and CPU-only physics simulation and camera rendering; using GPU rendering or running on real robots avoids this CPU simulation overhead and further reduces execution time, as detailed in A2 and A3 below. OpenRUA Goal-Position runs use Claude Code v2.1.310 (Opus 5.5 requires ≥v2.1.280; Table 1 used v2.1.226).

The results show that on Opus 5, our starter docs and tools (337 lines, 3,528 words) provide helpful initial guidance, reducing exploration time (59.3 to 55.8 min) and improving success (88.0% to 91.0%). However, once we upgrade to the stronger Opus 5.5, even without any starter docs or tools (0 lines, 856 words), our core workspace design alone reaches 91.0% and cuts mean episode time from 59.3 to 21.5 minutes. Taken together with ASPIRE (97.0% using 5,391 lines and 24,565 words), these results show that the modest extra gains from additional harness effort may quickly fade away as models continue to improve.

Therefore, the contribution of OpenRUA lies in finding a simple, general meta-method (native ROS 2 terminal access + a workspace) that allows agents to scale with model capability, rather than engineering complex harnesses from human priors to patch weaker models. In the words of The Bitter Lesson, our goal is to enable "AI agents that can discover like we can, not which contain what we have discovered."

AuthorsOpenReview ↗

Response to Reviewer 73qq (Part 3/5): Execution Time vs. Success Trade-Off (A2)


▶ W2: Insufficient analysis of the trade-off between execution time and success rate. Figure 6 and Appendix E analyze agent responses time and computational costs, but do not quantify the trade-off between actual task execution time and success. Moreover, simulation pauses during agent deliberation, excluding the effects of inference delays on a continuously evolving environment. It therefore remains unclear whether the reported success rates are achievable within practically useful time budgets.

A2: We add a trade-off analysis of actual task execution time versus success across both simulation and real-robot experiments, showing that high success rates are achievable within practical time budgets.

First, we measure how many tasks can be completed if the agent is only given a fixed wall-clock time budget (≤15, ≤30, ≤45, or ≤60 minutes), using the 100-episode LIBERO-PRO Goal-Position runs from A1 (Claude Code, high reasoning effort, w/o docs/tools). Here, time is the end-to-end active wall-clock time per episode, which includes model reasoning, code execution, and CPU-only physics simulation and multi-camera (640x480 RGB-D) rendering without a GPU; simulation pauses during deliberation to follow standard evaluation protocols (such as CaP-X and ASPIRE). Each column reports the percentage of all 100 episodes solved within that time budget:

Model (mean time/episode)≤15 min≤30 min≤45 min≤60 min
Opus 5 (59.3 min)11.0%45.0%67.0%71.0%
Opus 5.5 (21.5 min)50.0%74.0%85.0%90.0%

The results show two clear trends: (1) most successful episodes finish well within short time budgets (with Opus 5.5, the agent already solves 50.0% of all tasks within 15 minutes and 74.0% within 30 minutes); (2) the episode time drops rapidly as models improve: simply upgrading from Opus 5 to Opus 5.5 reduces mean episode time by over 60% (from 59.3 to 21.5 min) without adding any harness module.

Second, we measure actual task execution time on a physical Franka Panda robot using Codex + GPT-6 Astra (high reasoning effort) under a 30-minute time limit per trial, where physical time continues during agent thinking. Here, end-to-end time is measured over successful trials and includes model reasoning, code execution, real camera observation, physical robot motion, and any recovery (detailed real-robot results are reported in A3):

TaskSuccessful trials timedMean time (range), min
Place a banana on a plate39.6 (8.7–10.8)
Place a lime in a blue cup313.1 (9.9–14.9)
Place a small orange cup inside a large blue cup314.0 (13.0–14.4)
Place a pink bottle inside a tape ring113.6
Move a fruit from the pink plate to the gray plate313.1 (10.6–17.1)
Put a peach in the basket215.1 (12.9–17.2)
Place a tape roll on the pink plate311.0 (10.5–11.4)
Put two fruits from the table in the basket0–
Put a peach in the blue cup, then place the cup on the gray plate319.8 (14.1–24.5)
Stand a pink bottle upright217.3 (15.8–18.8)
Overall238.7–24.5

The results show that on a real robot, end-to-end task execution takes 8.7 to 24.5 minutes across all 23 successful trials (even including multi-step manipulation and online recovery such as re-grasping), confirming that OpenRUA is able to complete robot tasks within practical time budgets when physical time runs continuously.

Moreover, we'd like to clarify the difference between model inference frequency and robot control frequency in OpenRUA. In practice, model inference delays do not disrupt real-time robot control for two main reasons. First, model thinking mainly happens between steps when the arm and tabletop objects stay still. Second, once the agent starts a local ROS 2 program, that program handles motion and sensor feedback locally at the robot's control rate, unaffected by model inference latency. In this case, the model plans at a high level while the control program executes at high frequency, so slow inference can still drive fast local control. This means the model does not have to stay in the low-level control loop, as long as the agent can plan ahead and a brief reaction delay is acceptable.

In conclusion, for standard tabletop manipulation, OpenRUA works reliably within practical time budgets, while extending it to high-speed dynamic environments (where objects keep moving rapidly during model thinking) remains an important future direction as noted in the Limitations of our original submission.

Reviewer_SEdAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors present OpenRUA, a workspace as harness framework for Robot Use Agents. The framework provides an off the shelf coding agent with native ROS 2 interfaces, robot documentation, and basic tools. The agent autonomously writes perception and control programs during task execution. The authors conduct zero shot simulation experiments on CaP Bench, LIBERO PRO, and RoboCasa365. They also analyze how the agent autonomously constructs perception, control, and feedback control tools.

Critical strengths
  1. The paper introduces a Robot Use Agent perspective. It enables an off the shelf coding agent to autonomously construct task specific perception, motion control, and feedback control programs through ROS 2. The framework does not require robot policy training.

  2. The experimental evaluation is relatively systematic. The authors conduct large scale zero shot evaluation on CaP Bench, LIBERO PRO, and RoboCasa365. They further analyze different coding agents, reasoning budgets, perception strategies, and control behaviors.

Critical weaknesses
  1. The main contribution of OpenRUA focuses on the workspace and harness design. However, the current experiments do not sufficiently isolate the effect of interface design from the capability of the underlying coding model. Therefore, the contribution of the proposed framework remains difficult to attribute clearly.

  2. The workspace provides detailed guidance for perception, action, verification, grasping, and troubleshooting. It also includes six starter tools. However, the paper lacks component ablations without guidance or starter tools. These experiments are needed to quantify the contribution of each scaffolding component.

  3. All experiments are conducted in simulation. The simulation also pauses during agent deliberation. Therefore, the framework is not validated under real sensor noise, inference latency, continuous environment dynamics, or physical safety constraints.

Questions for authors

Please refer to the Critical Weaknesses.

6 comments
AuthorsOpenReview ↗

Response to Reviewer SEdA (Part 1/4): Matched-Model Comparison (A1)

Thank you for recognizing our training-free Robot-Use Agent perspective and our systematic zero-shot evaluation across three benchmarks. If we understood you correctly, your central concerns were three: 1) unclear interface attribution, 2) missing workspace ablations, and 3) simulation-only evaluation. We have addressed all three, and the result has strengthened our work. Specifically, in this rebuttal and the revised paper (new content in blue), we add matched-model comparisons (A1; Section 4.6), workspace ablations across model generations (A2; Section 4.6), and zero-shot physical-robot experiments (A3; Sections 4.7 and 5, Appendix K).

 


▶ W1: The main contribution of OpenRUA focuses on the workspace and harness design. However, the current experiments do not sufficiently isolate the effect of interface design from the capability of the underlying coding model. Therefore, the contribution of the proposed framework remains difficult to attribute clearly.

A1: We'd like to clarify that OpenRUA does not aim to build a complex harness method stacking specialized modules on top of coding agents for extra gains. Instead, following the spirit of the Bitter Lesson, our contribution lies in explicitly setting out to disintermediate or minimize the harness layer between the agent and the robot via simple ROS 2 terminal access plus a thin workspace. Our aim is to prompt the community to rethink current trends toward complex, specialized harness engineering as models grow more capable. To support this position, we add experimental evidence below.

To isolate the effect of this interface design under the same model capability, we compare OpenRUA with a strong baseline, ASPIRE, using its latest published results on the same Claude Code + Opus 5 (high reasoning effort) backbone across CaP-Bench and LIBERO-PRO (all 8 cells). To give a clear view of the additional harness effort in both methods, we use harness code (lines) and harness doc/prompt (words) as rough proxies; here, code refers to the pre-built Python tools, custom APIs, and workflow scripts supplied around the agent, and doc/prompt refers to the guide docs, workflow prompts, and machine descriptions provided before execution. Both methods are evaluated under each benchmark's official success predicates and paused-clock, sensor-only rules (no privileged object poses or test-time oracle access; initial-state protocols in Appendix B.1). In addition, OpenRUA runs zero-shot from a fresh workspace in every episode (100 trials/task on CaP-Bench; official initial states 0–9/task on LIBERO-PRO), whereas ASPIRE runs with task-specific skill experience accumulated beforehand in additional episodes.

MethodRobot interfaceEvaluation paradigmCaP-Bench ↑LIBERO-PRO ↑Harness code ↓Harness doc/prompt ↓
ASPIREStructured primitivesWith prior experience (skill library)98.86%94.45%5,391 lines24,565 words
OpenRUA (ours)Native ROS 2 + thin starter workspaceZero-shot (fresh workspace per episode)99.00%87.00%337 lines (16× less)3,528 words (7× less)

The results show that, under the same Opus 5 model, OpenRUA uses 16x less harness code (337 vs. 5,391 lines) and 7x less doc/prompt text (3,528 vs. 24,565 words) than ASPIRE, while matching it on CaP-Bench (99.00% vs. 98.86%) and reaching competitive zero-shot performance on LIBERO-PRO (87.00% vs. 94.45%). Here, ASPIRE's higher LIBERO-PRO score is consistent with its pre-evaluation experience. While OpenRUA can also carry experience across episodes in its workspace, we intentionally evaluate zero-shot to show that without prior experience or a heavy harness, native ROS 2 access and minimal starter materials already unlock a strong model's potential. To further demonstrate this point, we remove even these starter docs and tools in our ablation in A2 below.

AuthorsOpenReview ↗

Response to Reviewer SEdA (Part 2/4): Workspace Ablation Across Models (A2)


▶ W2: The workspace provides detailed guidance for perception, action, verification, grasping, and troubleshooting. It also includes six starter tools. However, the paper lacks component ablations without guidance or starter tools. These experiments are needed to quantify the contribution of each scaffolding component.

A2: Building on A1, we'd like to further highlight that these guidance docs and starter tools are not the core of OpenRUA by adding an ablation study: even when we remove all of them at once, the agent can still operate the robot from native ROS 2 access alone. Instead, our key point is that extra harness guidance may not be essential, and the need for it quickly fades as models evolve.

Specifically, we conduct this ablation on the LIBERO-PRO Goal-Position suite (10 tasks, official initial states 0–9; 100 episodes per setting), removing every starter document and tool while retaining only the core workspace design (native ROS 2 terminal access in a project directory plus the machine.yaml configuration). We evaluate this setting across both Opus 5 and the stronger Opus 5.5 under the same Claude Code (high reasoning effort) setup and identical evaluation rules:

MethodModelEvaluation paradigmSuccess ↑Mean time* ↓Harness code ↓Harness doc/prompt ↓
ASPIREOpus 5With prior experience97.0%–5,391 lines24,565 words
OpenRUAOpus 5Zero-shot91.0%55.8 min337 lines3,528 words
OpenRUA w/o docs/toolsOpus 5Zero-shot88.0%59.3 min0 lines856 words
OpenRUA w/o docs/toolsOpus 5.5Zero-shot91.0%21.5 min0 lines856 words

*Note: Mean time is the end-to-end active wall-clock time per episode, including model reasoning, code execution, and CPU-only physics simulation and camera rendering; using GPU rendering or running on real robots avoids this CPU simulation overhead and further reduces execution time, as reported in A3. OpenRUA Goal-Position runs use Claude Code v2.1.310 (Opus 5.5 requires ≥v2.1.280; Table 1 used v2.1.226).

The results show that on Opus 5, removing all guidance docs and starter tools together reduces success by only 3.0% (91.0% to 88.0%) and slightly increases mean time (55.8 to 59.3 min), showing that these starter docs/tools are not critical to task success. Moreover, once we upgrade to the stronger Opus 5.5, this bare workspace (0 lines, 856 words) reaches 91.0% and cuts mean episode time by over 60% (59.3 to 21.5 min). Taken together with ASPIRE (97.0% using 5,391 lines and 24,565 words), these results show that the modest extra gains from additional harness effort may quickly fade away as models continue to improve.

Therefore, our starter docs and tools act only as lightweight initial guidance for weaker models, whereas the core contribution of OpenRUA lies in finding a simple meta-method (native ROS 2 terminal access + a workspace) that scales directly with model capability.

AuthorsOpenReview ↗

Response to Reviewer SEdA (Part 3/4): Real-Robot Evaluation (A3a)


▶ W3: All experiments are conducted in simulation. The simulation also pauses during agent deliberation. Therefore, the framework is not validated under real sensor noise, inference latency, continuous environment dynamics, or physical safety constraints.

A3: You and the other reviewers suggested that we evaluate OpenRUA using an actual robot. We agree that such an experiment is essential, so we add real-robot experiments on a Franka Panda across 10 diverse manipulation tasks following task types in OpenVLA and π0 (3 trials per task, spanning plate and container placement, cup nesting, bottle uprighting, and multi-step manipulation), with videos and execution traces of all 30 trials available at https://anon-openrua.github.io. We highlight that OpenRUA is built on standard ROS 2 and does not rely on anything specific to simulation, so it can naturally carry over to real robots without any additional work.

First, we run OpenRUA using Codex + GPT-6 Astra (high reasoning effort), where physical time continues during agent thinking, and the agent reads real RGB-D camera inputs, writes ROS 2 programs, and executes them on the robot. To test generality under a strict real-world setting, each trial starts zero-shot from a fresh workspace and session with a 30-minute time limit and no code or experience carried over from earlier trials. Across the 3 trials per task, we vary not only the initial tabletop layout, but also the robot arm's starting position and orientation. In the table below, end-to-end time is measured over successful trials and includes model reasoning, code execution, real camera observation, physical robot motion, and any recovery:

TaskSuccess / trialsMean time (range), min
Place a banana on a plate3/39.6 (8.7–10.8)
Place a lime in a blue cup3/313.1 (9.9–14.9)
Place a small orange cup inside a large blue cup3/314.0 (13.0–14.4)
Place a pink bottle inside a tape ring1/313.6
Move a fruit from the pink plate to the gray plate3/313.1 (10.6–17.1)
Put a peach in the basket2/315.1 (12.9–17.2)
Place a tape roll on the pink plate3/311.0 (10.5–11.4)
Put two fruits from the table in the basket0/3–
Put a peach in the blue cup, then place the cup on the gray plate3/319.8 (14.1–24.5)
Stand a pink bottle upright2/317.3 (15.8–18.8)
Overall23/30 (76.7%)8.7–24.5

The results show that OpenRUA achieves 23/30 (76.7%) overall success across the 10 diverse physical tasks within 8.7 to 24.5 minutes per successful trial, confirming that the generated ROS 2 programs can reliably complete real-world manipulation tasks when physical time runs continuously. More importantly, this validates OpenRUA's minimalist design: exposing standard ROS 2 alone is enough for the agent to handle real-world timing and sensor noise, without adding a single line of sim-to-real engineering code.

Reviewer_BmVpOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OpenRUA investigates whether an off-the-shelf coding agent can serve as a zero-shot visuomotor policy through a robot’s native ROS 2 interface, without learned robot policies or task-specific primitive libraries. The agent writes and executes its own perception and control programs, using execution feedback to refine its behavior. The paper reports 99.0% success on CaP-Bench and 87.0% on LIBERO-PRO, alongside analyses of the perception and control programs generated during execution. The work offers an interesting alternative to increasingly specialized robot-agent architectures.

Critical strengths
  1. The paper challenges the assumption that robot-use agents require extensive bespoke abstractions and predefined workflows.
  2. The workspace-as-harness approach is easy to understand and potentially generalizable beyond robotics.
  3. .Performance across three simulation benchmarks provides meaningful evidence that coding agents can perform diverse manipulation tasks without task-specific learned policies.
  4. Examining agent-generated perception and control programs provides insight beyond aggregate success rates.
Critical weaknesses
  1. Comparisons with prior systems do not clearly establish whether performance gains result from the minimalist interface, the underlying coding model, or differences in evaluation conditions. A controlled comparison would strengthen the claim of the paper.
  2. The frequency of self-written perception and feedback-control programs demonstrates their use, but does not establish how much they contribute to task success.
  3. All experiments are conducted in simulation, and performance on unseen composite tasks remains limited. The results therefore do not establish robust real-world robot control.
Questions for authors
  1. Can the authors compare the proposed interface against a structured or primitive-based alternative using the same model and comparable execution budgets? It would also be useful to evaluate on LIBERO-PRO, since RoboSuite may be too saturated to clearly distinguish between methods.
  2. Do controlled ablations demonstrate that self-written perception and feedback-control programs improve task success?
  3. How robust are the reported results across random seeds, initial states, and different coding-agent models? I’m concerned that the agent may be able to debug on the same seeds used for evaluation, which could inflate the reported success rate. For comparison, in our ASPIRE setup, we use seeds 101–125 for debugging and seeds 1–100 for evaluation on RoboSuite.
7 comments
AuthorsOpenReview ↗

Response to Reviewer BmVp (Part 1/4): Matched-Model Comparison (A1)

We appreciate your recognition that our simple, generalizable workspace-as-harness challenges the need for bespoke abstractions and predefined workflows. It is also encouraging that you find value in examining agent-generated perception and control programs.

If we understood you correctly, your central concerns were two: 1) that we lacked controlled comparisons and ablations to isolate our interface's contribution, and 2) that our evaluation relied solely on simulation. We have addressed both. The result has greatly strengthened our work. Specifically, in this rebuttal and the revised paper (new content in blue), we add a matched-model comparison on LIBERO-PRO (A1; Section 4.6), analyze how self-written programs contribute to task success (A2; Section 4.6 and Appendix J), report new real-robot experiments alongside composite-task results (A3; Section 4.7 and Appendix K), and clarify our zero-shot evaluation setup across seeds and models (A4).

 


▶ W1 & Q1: Comparisons with prior systems do not clearly establish whether performance gains result from the minimalist interface, the underlying coding model, or differences in evaluation conditions. A controlled comparison would strengthen the claim of the paper. / Can the authors compare the proposed interface against a structured or primitive-based alternative using the same model and comparable execution budgets? It would also be useful to evaluate on LIBERO-PRO, since RoboSuite may be too saturated to clearly distinguish between methods.

A1: Thank you for recognizing our minimalist design (S1), which follows the spirit of the Bitter Lesson: finding a simple, general meta-method (native ROS 2 terminal access + a thin workspace) for "AI agents that can discover like we can", allowing robot control to scale directly with model capability. To separate the effects of interface design, evaluation conditions, and model strength on LIBERO-PRO, we report a controlled comparison below.

Following your suggestion to evaluate on LIBERO-PRO, we compare OpenRUA with a strong structured, primitive-based baseline, ASPIRE, using its latest published results on the same Claude Code + Opus 5 (high reasoning effort) backbone across all 80 tasks (all 8 cells). Both systems are evaluated under the benchmark's official success predicates in paused-clock simulation from sensor observations alone, with no privileged object poses or test-time oracle access. Their main difference lies in prior experience (see A4): ASPIRE accumulates experience into a skill library before evaluation (seeds 51–65 for experience accumulation, seeds 1–50 for evaluation). In contrast, OpenRUA solves every episode zero-shot from a fresh workspace with no prior experience (official initial states 0–9, up to 500 turns per episode; see Appendix B.2, Table 6).

MethodRobot interfaceEvaluation paradigmLIBERO-PRO ↑
ASPIREStructured primitivesWith prior experience (skill library)94.45%
OpenRUA (ours)Native ROS 2 + thin starter workspaceZero-shot (fresh workspace per episode)87.00%

The results show that under the same Opus 5 model, ASPIRE's higher LIBERO-PRO score (94.45%) is consistent with its pre-evaluation experience over structured primitives, which requires designing domain-specific primitives and collecting prior rollouts for each task. While OpenRUA can also carry experience across episodes in its workspace, we intentionally evaluate zero-shot to show that even without any custom primitives or prior experience, OpenRUA already solves 87.00% on the first try over native ROS 2 alone. More importantly, keeping the interface minimal not only enables direct zero-shot transfer to unseen tasks and real robots (A3), but also lets performance scale naturally as stronger models emerge (Table 6).

AuthorsOpenReview ↗

Response to Reviewer BmVp (Part 2/4): Contribution of Self-Written Programs (A2)


▶ W2 & Q2: The frequency of self-written perception and feedback-control programs demonstrates their use, but does not establish how much they contribute to task success. / Do controlled ablations demonstrate that self-written perception and feedback-control programs improve task success?

A2: We add a controlled workspace ablation and a trajectory-level analysis below, showing that self-written perception and feedback-control programs directly contribute to task success.

First, we remove all starter documentation and tools to isolate the effect of self-written programs (OpenRUA w/o docs/tools), leaving 0 lines of starter code and only the machine.yaml robot description over native ROS 2. We evaluate this setting across Opus 5 and Opus 5.5 on the LIBERO-PRO Goal-Position suite (10 tasks, official initial states 0–9; 100 episodes per setting under Claude Code with high reasoning effort and identical evaluation rules; these Goal-Position runs use v2.1.310 as Opus 5.5 requires ≥v2.1.280, while Table 1 used v2.1.226):

MethodModelRobot interfaceSuccess ↑Starter code ↓
OpenRUAOpus 5Native ROS 2 + thin starter workspace91.0%337 lines
OpenRUA w/o docs/toolsOpus 5Native ROS 2 only (machine.yaml)88.0%0 lines
OpenRUA w/o docs/toolsOpus 5.5Native ROS 2 only (machine.yaml)91.0%0 lines

The results show that even after removing all starter docs and tools (0 lines of pre-built code), OpenRUA still reaches 88.0% on Opus 5 (only a 3.0% drop from 91.0%) and 91.0% on Opus 5.5. Since no pre-built tools are provided in this setting, the agent has to write all perception and control scripts on its own, confirming that task success comes directly from self-written programs rather than starter helpers.

Second, to further show the roles of self-written perception and feedback-control programs, we inspect all 91 successful trajectories of OpenRUA w/o docs/tools (Opus 5.5) to check where 3D target coordinates come from and how feedback control acts during execution:

  • Self-written perception programs: Because no ground-truth object poses are provided, in all 91/91 (100%) successful episodes the agent relies on its own perception scripts (e.g., RGB-D pixel-to-3D projection) to compute 3D target positions from raw camera and depth data. Across all 378 motions targeting scene objects (grasping, placing, pushing, or operating handles and knobs), 368 (97.4%) get their 3D coordinates from these self-written scripts.
  • Self-written feedback-control programs: In 50/91 (54.9%) successful episodes, the agent's control scripts actively catch motion errors or contact forces and adjust commands on their own, such as retrying incomplete moves from live joint errors until convergence or stopping descent on contact. In 14/91 (15.4%) episodes, these automatic retries directly prevent grasp or placement failures when a move stops 0.3–0.8 rad short before descending or operating the gripper.

Together, these results confirm that 3D perception and closed-loop control remain essential for manipulation: self-written perception scripts supply nearly every 3D target, while feedback-control scripts actively correct local execution errors alongside visual recovery between turns. Yet even with zero starter code, a strong coding agent can already write both on its own over native ROS 2, showing that capabilities we usually hand-engineer can increasingly be discovered by the agent itself.

AuthorsOpenReview ↗

Response to Reviewer BmVp (Part 3/4): Real-Robot & Composite-Task Results (A3)


▶ W3: All experiments are conducted in simulation, and performance on unseen composite tasks remains limited. The results therefore do not establish robust real-world robot control.

A3: You and the other reviewers suggested that we evaluate OpenRUA using an actual robot. We agree that such an experiment is essential, so we add real-robot experiments on a Franka Panda across 10 diverse manipulation tasks following task types in OpenVLA and π0 (3 trials per task, spanning plate and container placement, cup nesting, bottle uprighting, and multi-step manipulation), with videos and execution traces of all 30 trials available at https://anon-openrua.github.io. We highlight that OpenRUA is built on standard ROS 2 and does not rely on anything specific to simulation, so it can naturally carry over to real robots without any additional work.

First, we run OpenRUA using Codex + GPT-6 Astra (high reasoning effort), where physical time continues during agent thinking, and the agent reads real RGB-D camera inputs, writes ROS 2 programs, and executes them on the robot. To test generality under a strict real-world setting, each trial starts zero-shot from a fresh workspace and session with a 30-minute time limit and no code or experience carried over from earlier trials. Across the 3 trials per task, we vary not only the initial tabletop layout, but also the robot arm's starting position and orientation. In the table below, end-to-end time is measured over successful trials and includes model reasoning, code execution, real camera observation, physical robot motion, and any recovery:

TaskSuccess / trialsMean time (range), min
Place a banana on a plate3/39.6 (8.7–10.8)
Place a lime in a blue cup3/313.1 (9.9–14.9)
Place a small orange cup inside a large blue cup3/314.0 (13.0–14.4)
Place a pink bottle inside a tape ring1/313.6
Move a fruit from the pink plate to the gray plate3/313.1 (10.6–17.1)
Put a peach in the basket2/315.1 (12.9–17.2)
Place a tape roll on the pink plate3/311.0 (10.5–11.4)
Put two fruits from the table in the basket0/3–
Put a peach in the blue cup, then place the cup on the gray plate3/319.8 (14.1–24.5)
Stand a pink bottle upright2/317.3 (15.8–18.8)
Overall23/30 (76.7%)8.7–24.5

The results show that OpenRUA achieves 23/30 (76.7%) overall success across the 10 diverse physical tasks within 8.7 to 24.5 minutes per successful trial, confirming that the generated ROS 2 programs can reliably complete real-world manipulation tasks when physical time runs continuously. More importantly, this validates OpenRUA's minimalist design: exposing standard ROS 2 alone is enough for the agent to handle real-world timing and sensor noise, without adding a single line of sim-to-real engineering code.

Second, we highlight representative real-robot cases below:

  • Autonomous perception and recovery in successful trials: In a lime-to-cup trial, the agent first pushed the lime aside and mistakenly grasped the plate edge. After seeing this in new camera images, it released the plate, adjusted its grasp height, and placed the lime into the blue cup. In the successful bottle-to-ring trial, it also wrote its own image and depth script and used multi-view observations to align and place the bottle upright.
  • Failure modes across the 7 unsuccessful trials: Most failures (4/7) came from grasp or release instability where a fruit or bottle slipped or rolled off the table and out of reach (e.g., when re-grasping a tipped bottle on the ring edge, or handling the second fruit after placing the first in the basket). Of the remaining three, one multi-fruit trial reached the 30-minute limit on the second fruit, one bottle-uprighting trial triggered a protective stop from a pulled camera cable, and one bottle-to-ring trial produced a large arm rotation during a 5.5 cm downward move (consistent with an IK configuration jump) and was stopped by the operator.

Finally, regarding unseen composite tasks, our physical suite above includes two multi-step tasks: OpenRUA solves Put a peach in the blue cup, then place the cup on the gray plate in 3/3 trials, while on Put two fruits from the table in the basket (0/3) it places the first fruit in 2 of the 3 trials before failing due to fruit slippage/roll-off or the 30-minute limit. In simulation on RoboCasa365 (Table 5), OpenRUA also shows no drop from seen to unseen composite tasks (26.3% vs. 28.1%, nearly doubling Harness VLA's 13.8%–15.0%). While longer-horizon manipulation still leaves room for improvement, multi-step reliability improves steadily as stronger models emerge (Table 6).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal and Discussion

Dear Area Chair and Reviewers,

The discussion phase highlighted two central concerns shared across all three reviews (BmVp, 73qq, SEdA): isolating OpenRUA's contribution beyond the underlying model, and validating real-world applicability beyond simulation. Over the past two weeks, we ran new experiments and analyses in our rebuttal and revised paper to address both directly, and Reviewer BmVp noted that these additions strengthen our central claim and raised their score:

  1. Isolating OpenRUA's contribution. To address concerns over how to attribute our results given model differences across baselines, we added matched-model comparisons against a structured primitive-based baseline and workspace ablations removing all starter documentation and tools. Together with trajectory-level audits, these experiments confirm that self-written perception and feedback-control programs directly drive task success, establishing native ROS 2 access plus a thin workspace as a simple, general meta-method (following the spirit of the Bitter Lesson) that scales with model capability rather than stacking complex modules on top of agents.
  2. Validating real-world applicability. To address concerns that paused-clock simulation did not verify continuous-time execution, latency, or physical safety, we evaluated OpenRUA zero-shot on a physical Franka Panda across 10 diverse manipulation tasks with zero sim-to-real code (23/30 successful trials; videos and execution traces at https://anon-openrua.github.io), paired with execution-time and safety analyses. This confirms that over standard ROS 2, coding agents can reliably handle real-world perception, task planning, and online recovery without specialized task harnesses.

Taken together, these additions resolve both shared concerns and aim to prompt a rethink of current trends toward complex harness engineering, freeing our community to focus on physical reliability and execution safety on real robots rather than specialized harnesses that stronger models will soon render obsolete.

We appreciate the feedback and engagement throughout the review process, which have helped us strengthen the paper.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.