- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
OpenRUA gives off-the-shelf coding agents terminal access to native ROS 2 interfaces and a workspace containing machine documentation and starter tools, allowing the agents to acquire observations and write and execute robot-control programs without a prescribed workflow. The main evaluation reports 99.0% success on CaP-Bench, 87.0% on LIBERO-PRO, and 39.4% on RoboCasa365. The paper additionally analyzes agent-written perception and control programs, alternative model configurations and computational costs, and failed episodes.
Critical strengths
- Extensive simulation evaluation. The paper evaluates tasks across CaP-Bench, LIBERO-PRO, and RoboCasa365, providing substantial empirical coverage of the system’s capabilities in simulation.
- The extensive code listings and explanations in the appendix, in my view, reflect the authors’ confidence in the authenticity of their work.
Critical weaknesses
- Unclear contribution beyond off-the-shelf agents. The baseline comparisons involve different underlying models: OpenRUA uses Opus 5, whereas ASPIRE uses Opus 4.6 and Harness VLA uses Opus 4.7 or GPT-5.5. These model differences make it difficult to attribute the reported performance gains to OpenRUA's harness rather than the underlying agents' capabilities. Without matched-model interface comparisons or workspace ablations, the harness's own contribution remains insufficiently established.
- Insufficient analysis of the trade-off between execution time and success rate. Figure 6 and Appendix E analyze agent responses time and computational costs, but do not quantify the trade-off between actual task execution time and success. Moreover, simulation pauses during agent deliberation, excluding the effects of inference delays on a continuously evolving environment. It therefore remains unclear whether the reported success rates are achievable within practically useful time budgets.
Besides, it is also lack of real-world validation. All experiments on CaP-Bench, LIBERO-PRO, and RoboCasa365 are conducted in simulation, with no physical-robot validation. The generated programs can reliably and safely complete tasks on physical robots remains unverified. This substantially limits the evidence for the system's practical applicability.
Questions for authors
In limitation part, you note that agents sometimes implement their own force thresholds and stopping conditions. When autonomously constructing their control programs, do agents also spontaneously build a more systematic safety framework, such as motion limits or collision checks?
7 comments
Response to Reviewer 73qq (Part 2/5): Workspace Ablation Across Models (A1b)
Second, we conduct a workspace ablation on the LIBERO-PRO Goal-Position suite (10 tasks, official initial states 0–9; 100 episodes per setting). Specifically, we retain the core workspace design, i.e., giving the agent native ROS 2 terminal access in a project directory, while removing all starter documentation and tools and leaving only the machine.yaml configuration. We evaluate this setting across both Opus 5 and the stronger Opus 5.5 under the same Claude Code (high reasoning effort) setup and identical evaluation rules.
| Method | Model | Evaluation paradigm | Success ↑ | Mean time* ↓ | Harness code ↓ | Harness doc/prompt ↓ |
|---|---|---|---|---|---|---|
| ASPIRE | Opus 5 | With prior experience | 97.0% | – | 5,391 lines | 24,565 words |
| OpenRUA | Opus 5 | Zero-shot | 91.0% | 55.8 min | 337 lines | 3,528 words |
| OpenRUA w/o docs/tools | Opus 5 | Zero-shot | 88.0% | 59.3 min | 0 lines | 856 words |
| OpenRUA w/o docs/tools | Opus 5.5 | Zero-shot | 91.0% | 21.5 min | 0 lines | 856 words |
*Note: Mean time is the end-to-end active wall-clock time per episode, including model reasoning, code execution, and CPU-only physics simulation and camera rendering; using GPU rendering or running on real robots avoids this CPU simulation overhead and further reduces execution time, as detailed in A2 and A3 below. OpenRUA Goal-Position runs use Claude Code v2.1.310 (Opus 5.5 requires ≥v2.1.280; Table 1 used v2.1.226).
The results show that on Opus 5, our starter docs and tools (337 lines, 3,528 words) provide helpful initial guidance, reducing exploration time (59.3 to 55.8 min) and improving success (88.0% to 91.0%). However, once we upgrade to the stronger Opus 5.5, even without any starter docs or tools (0 lines, 856 words), our core workspace design alone reaches 91.0% and cuts mean episode time from 59.3 to 21.5 minutes. Taken together with ASPIRE (97.0% using 5,391 lines and 24,565 words), these results show that the modest extra gains from additional harness effort may quickly fade away as models continue to improve.
Therefore, the contribution of OpenRUA lies in finding a simple, general meta-method (native ROS 2 terminal access + a workspace) that allows agents to scale with model capability, rather than engineering complex harnesses from human priors to patch weaker models. In the words of The Bitter Lesson, our goal is to enable "AI agents that can discover like we can, not which contain what we have discovered."
Response to Reviewer 73qq (Part 3/5): Execution Time vs. Success Trade-Off (A2)
▶ W2: Insufficient analysis of the trade-off between execution time and success rate. Figure 6 and Appendix E analyze agent responses time and computational costs, but do not quantify the trade-off between actual task execution time and success. Moreover, simulation pauses during agent deliberation, excluding the effects of inference delays on a continuously evolving environment. It therefore remains unclear whether the reported success rates are achievable within practically useful time budgets.
A2: We add a trade-off analysis of actual task execution time versus success across both simulation and real-robot experiments, showing that high success rates are achievable within practical time budgets.
First, we measure how many tasks can be completed if the agent is only given a fixed wall-clock time budget (≤15, ≤30, ≤45, or ≤60 minutes), using the 100-episode LIBERO-PRO Goal-Position runs from A1 (Claude Code, high reasoning effort, w/o docs/tools). Here, time is the end-to-end active wall-clock time per episode, which includes model reasoning, code execution, and CPU-only physics simulation and multi-camera (640x480 RGB-D) rendering without a GPU; simulation pauses during deliberation to follow standard evaluation protocols (such as CaP-X and ASPIRE). Each column reports the percentage of all 100 episodes solved within that time budget:
| Model (mean time/episode) | ≤15 min | ≤30 min | ≤45 min | ≤60 min |
|---|---|---|---|---|
| Opus 5 (59.3 min) | 11.0% | 45.0% | 67.0% | 71.0% |
| Opus 5.5 (21.5 min) | 50.0% | 74.0% | 85.0% | 90.0% |
The results show two clear trends: (1) most successful episodes finish well within short time budgets (with Opus 5.5, the agent already solves 50.0% of all tasks within 15 minutes and 74.0% within 30 minutes); (2) the episode time drops rapidly as models improve: simply upgrading from Opus 5 to Opus 5.5 reduces mean episode time by over 60% (from 59.3 to 21.5 min) without adding any harness module.
Second, we measure actual task execution time on a physical Franka Panda robot using Codex + GPT-6 Astra (high reasoning effort) under a 30-minute time limit per trial, where physical time continues during agent thinking. Here, end-to-end time is measured over successful trials and includes model reasoning, code execution, real camera observation, physical robot motion, and any recovery (detailed real-robot results are reported in A3):
| Task | Successful trials timed | Mean time (range), min |
|---|---|---|
| Place a banana on a plate | 3 | 9.6 (8.7–10.8) |
| Place a lime in a blue cup | 3 | 13.1 (9.9–14.9) |
| Place a small orange cup inside a large blue cup | 3 | 14.0 (13.0–14.4) |
| Place a pink bottle inside a tape ring | 1 | 13.6 |
| Move a fruit from the pink plate to the gray plate | 3 | 13.1 (10.6–17.1) |
| Put a peach in the basket | 2 | 15.1 (12.9–17.2) |
| Place a tape roll on the pink plate | 3 | 11.0 (10.5–11.4) |
| Put two fruits from the table in the basket | 0 | – |
| Put a peach in the blue cup, then place the cup on the gray plate | 3 | 19.8 (14.1–24.5) |
| Stand a pink bottle upright | 2 | 17.3 (15.8–18.8) |
| Overall | 23 | 8.7–24.5 |
The results show that on a real robot, end-to-end task execution takes 8.7 to 24.5 minutes across all 23 successful trials (even including multi-step manipulation and online recovery such as re-grasping), confirming that OpenRUA is able to complete robot tasks within practical time budgets when physical time runs continuously.
Moreover, we'd like to clarify the difference between model inference frequency and robot control frequency in OpenRUA. In practice, model inference delays do not disrupt real-time robot control for two main reasons. First, model thinking mainly happens between steps when the arm and tabletop objects stay still. Second, once the agent starts a local ROS 2 program, that program handles motion and sensor feedback locally at the robot's control rate, unaffected by model inference latency. In this case, the model plans at a high level while the control program executes at high frequency, so slow inference can still drive fast local control. This means the model does not have to stay in the low-level control loop, as long as the agent can plan ahead and a brief reaction delay is acceptable.
In conclusion, for standard tabletop manipulation, OpenRUA works reliably within practical time budgets, while extending it to high-speed dynamic environments (where objects keep moving rapidly during model thinking) remains an important future direction as noted in the Limitations of our original submission.
Response to Reviewer 73qq (Part 1/5): Contribution & Matched-Model Comparison (A1a)
Thank you for recognizing our extensive benchmark evaluation. If we understood you correctly, your central concerns were two: 1) that our harness's contribution was unclear, and 2) that our evaluation lacked real-world validation. We have addressed both, and the result has strengthened our work. Specifically, in this rebuttal and the revised paper (new content in blue), we add matched-model comparisons and workspace ablations (A1; Section 4.6), alongside time-budget analyses, physical-robot experiments, and safety evaluations (A2–A4; Sections 4.7 and 5, Appendix K).
▶ W1: Unclear contribution beyond off-the-shelf agents. The baseline comparisons involve different underlying models: OpenRUA uses Opus 5, whereas ASPIRE uses Opus 4.6 and Harness VLA uses Opus 4.7 or GPT-5.5. These model differences make it difficult to attribute the reported performance gains to OpenRUA's harness rather than the underlying agents' capabilities. Without matched-model interface comparisons or workspace ablations, the harness's own contribution remains insufficiently established.
A1: Our contribution lies precisely in showing that, beyond off-the-shelf agents, robot manipulation does not have to rely on stacking complex modules or performance tricks on top of them for extra gains. Instead, following the spirit of the Bitter Lesson, we explicitly set out to disintermediate or minimize the harness layer between the agent and the robot via simple ROS 2 terminal access plus a thin workspace. Our aim is to prompt the community to rethink current trends toward complex, specialized harness engineering as models grow more capable. To support this position, we add experimental evidence below.
First, we compare OpenRUA with a strong baseline, ASPIRE, using its latest published results on the same Claude Code + Opus 5 (high reasoning effort) backbone across CaP-Bench and LIBERO-PRO (all 8 cells). To give a clear view of the additional harness engineering effort in both methods, we use harness code (lines) and harness doc/prompt (words) as rough proxies; here, code refers to the pre-built Python tools, custom APIs, and workflow scripts directly supplied around the agent, and doc/prompt refers to the guide docs, workflow prompts, and machine descriptions provided before task execution. Both methods are evaluated under each benchmark's official success predicates and paused-clock, sensor-only rules (no privileged object poses or test-time oracle access; initial-state protocols in Appendix B.1). In addition, OpenRUA runs zero-shot from a fresh workspace in every episode (100 trials/task on CaP-Bench; official initial states 0–9/task on LIBERO-PRO), whereas ASPIRE runs with task-specific skill experience accumulated beforehand in additional episodes.