acceptodds
Under review as a conference paper at ICLR 2027

RobotUse: Allocating Computation, Context, and Decisions

Abstract

Robot agents must connect their intended actions to observed outcomes while retaining the context needed to revise their choices over repeated attempts. Existing interfaces often leave these choices inside predefined tools or require agents to manage detailed execution code and its growing history. We introduce RobotUse, a robot agent harness that organizes computation, context, and decisions around specifying and revising physical actions. Agents visually select targets and poses, while the backend handles geometry, motion planning, and control. Subagents retain detailed interactions within each subgoal and return the information needed for subsequent decisions. Continual harnessing lets agents learn from execution by updating a persistent playbook. On RoboLab, RobotUse achieves 45% task success, outperforming CaP-X by 6.7 percentage points while maintaining compact decision contexts and reducing reliance on predefined action abstractions. Furthermore, we show that RobotUse learns from real-world execution despite imperfect feedback and transfers what it learns to subsequent tasks. Project page is available at https://anonrobotuse.github.io/RobotUse/.

Primary area:
applications to robotics, autonomy, planning
Keywords:
robot manipulation, robot harness, context engineering, continual harness, foundation model agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_gtfTOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents RobotUse, a robot-agent harness that separates task-level decision-making, visual action specification, and low-level robot execution. The framework employs a main agent for task planning, subagents for visually selecting and refining spatial actions, and a robot backend for geometric computation, motion planning, and control. It further introduces subgoal-local contexts to reduce context overhead and a persistent playbook that is updated from execution experience.

Experiments on RoboLab show that RobotUse achieves a 45.0% task success rate, outperforming CaP-X by 6.67 percentage points. The authors also evaluate context efficiency, performance improvements through playbook refinement, and execution on three real-robot manipulation tasks.

Critical strengths

The paper addresses a practical problem in organizing robot-agent execution. The separation of task-level reasoning, local spatial decisions, and backend motion control is clearly presented. The context-handoff ablation also provides useful evidence that isolating subgoal-specific interaction histories can substantially reduce token consumption compared with a single-agent implementation.

Critical weaknesses

1. Limited methodological novelty and insufficient justification of the cost–performance trade-off.

The proposed framework largely combines existing concepts, including multi-agent delegation, visual action selection, context isolation, and persistent playbook refinement. While these components form a functional system, the paper does not convincingly establish what fundamentally new methodology or insight RobotUse contributes beyond their integration.

Moreover, the performance improvement over CaP-X is relatively modest (45.0% vs. 38.33%), despite requiring approximately three times the API cost and execution time, and nearly five times the token consumption. The compared methods also use different execution budgets and stopping criteria. Without comparisons under matched resource constraints, it is difficult to determine whether the additional complexity and computational cost of RobotUse are justified.

2. Insufficient evidence for generalizable learning and real-world effectiveness.

The continual harnessing experiments repeatedly refine and evaluate the playbook on the same 40 RoboLab tasks. Although the final evaluation uses new episodes, the task set remains unchanged. As a result, the reported improvement from 47.5% to 55.0% may primarily reflect adaptation to recurring task-specific failures rather than transferable knowledge.

The real-robot evaluation is also limited to three relatively simple manipulation tasks. Importantly, the authors refine the playbook until an initial success before conducting the subsequent evaluations. Therefore, the reported 100% success rates reflect performance after task-specific adaptation rather than zero-shot generalization. The adaptation cost is not sufficiently characterized, and the experiments provide limited evidence that RobotUse can reliably handle more complex, long-horizon manipulation tasks.

Questions for authors

Q1. Methodological novelty compared with existing agent harnesses.

The proposed framework appears to build primarily on established techniques such as subagent delegation, visual action selection, and persistent experience-based guidance. Could the authors clarify what distinguishes RobotUse from existing approaches such as PIVOT and Continual Harness? What specific methodological contribution or new insight does this framework provide beyond combining these components?

Q2. Fair comparison under matched computational budgets.

RobotUse achieves a relatively modest improvement over CaP-X while requiring substantially more API cost, tokens, and execution time. Additionally, the compared methods use different execution budgets and stopping criteria. Could the authors provide comparisons under matched execution time, API cost, or interaction budgets? This would help determine whether RobotUse offers a meaningful advantage beyond simply allocating more resources to each task.

Q3. Generalization of continual playbook refinement.

The playbook is repeatedly refined and evaluated on the same 40 RoboLab tasks, making it difficult to distinguish generalizable learning from adaptation to task-specific failure patterns. Could the authors evaluate the refined playbook on a held-out set of tasks that were never used during refinement? Such an experiment would provide stronger evidence that RobotUse learns transferable strategies rather than becoming increasingly specialized to the evaluation task catalog.

Q4. Real-robot evaluation and adaptation cost.

The real-robot experiments cover only three basic manipulation tasks, and the playbook is refined until an initial success before the reported evaluations. Could the authors clarify the number of attempts, execution time, and computational cost required during this adaptation process? It would also be valuable to compare methods under equivalent adaptation opportunities and evaluate RobotUse on more challenging, multi-stage manipulation tasks. This would help establish whether the proposed harness provides practical benefits beyond reliable repetition of relatively simple, previously adapted tasks.

Reviewer_pwqsOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents RoboUse, a agent harness that organizes computation, context, and decisions. It designs visual action specification, subagent-based context management, and persistent playbook refinement to specify and revise physical actions while delegating geometric computation and motion planning to a backend. The framework is evaluated on both simulation and real-robot tasks, showcasing promising task performance and the ability to improve execution through accumulated experience.

Critical strengths
  1. A practical design with clear motivation. The paper explores how to allocate decision-making, computation, context, and execution experience in robot agents. The design is intuitive and allows agents to learn from execution feedback through persistent playbook refinement, without updating model parameters or modifying backend code.
  2. The paper evaluates the framework across a diverse set of simulation and real-robot tasks. The visualizations clearly illustrate the interaction process.
Critical weaknesses
  1. The contributions of individual components are not sufficiently isolated. Although the paper provides some ablation studies, it remains unclear how much each component contributes to the overall performance improvements.
  2. The real-robot evaluation is limited, and the contribution of playbook refinement is unclear. The experiments cover only three relatively simple manipulation tasks. Although RobotUse achieves 100% success in the evaluation trials after task-specific playbook refinement, the paper does not report a controlled comparison without a playbook or with the initial, unrefined playbook. Therefore, the improvement attributable to playbook refinement is difficult to quantify.
  3. The execution efficiency remains a concern. RobotUse appears to incur substantial overhead in task completion time. Since the framework involves multiple rounds of agent reasoning, tool interactions, and execution feedback, it would be useful to better characterize its latency and identify the main sources of overhead.
Questions for authors

As per the above weaknesses, I have the following questions:

  1. Could the authors provide more controlled ablations to isolate the contributions of the key components?
  2. Could the authors compare real-world performance with and without playbook refinement and evaluate more challenging tasks?
  3. What are the main sources of runtime overhead, and how can task execution be made more efficient?
Reviewer_SnCUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes RobotUse, a framework that separates task planning, visual action selection, and robot execution. The main agent generates subgoals, while subagents select targets, inspect grasp candidates, and adjust poses from visual feedback. The backend handles geometry and robot control. Detailed interactions stay within the subagent context, and a persistent text playbook stores guidance from previous execution. Experiments cover 40 RoboLab tasks and 3 real-robot tasks. RobotUse achieves better simulation success, reduces token usage compared with a single-agent implementation, and shows improvements after playbook refinement.

Critical strengths
  • The separation between task planning, local visual interaction, and robot execution makes sense. The agent can inspect and adjust the proposed grasp instead of directly accepting the backend’s choice. The examples show how this helps recover from imperfect tool outputs.

  • The context ablation is useful. With the same model, tools, playbook, and backend, separating local interactions largely reduces total token usage, while success increases by about 5%. This provides some evidence that the context organization helps reduce inference cost and generates a better action decision.

Critical weaknesses
  • The major issue of the paper is that I do not clearly see how much of the performance improvement comes from the proposed action interface. For example. RobotUse achieves 45% compared with 38.33% for CaP-X, but it uses about three times the reported API cost and elapsed time, and with different stopping rules. There is also no controlled ablation removing visual candidate inspection or pose adjustment. The grasp candidates still depend on the backend’s perception and grasp-generation tools. Partial observations or occlusion may give incomplete geometry, and a visually reasonable grasp can still slip because of uncertain physical properties. The examples are useful, but they do not establish how reliably visual adjustment handles these cases. I would like to understand whether the improvement comes from the interface, the backend, or more opportunities to retry.

  • The evidence for learning from experience and transfer seems limited. The adaptation mainly updates a text playbook while keeping the model and backend fixed. This can be useful, but simulation success only increases from 19/40 to 22/40 on the same task catalog used for refinement. The real-robot results are good, but only cover three quasi-static tasks in one sequential adaptation campaign. Moreover, based on “carrying the playbook over to stacking a cube on another cube yields success on the first attempt.”, stacking succeeds on the first attempt, which could also happen without previous experience. For button pressing, one failure followed by one success does not isolate the effect of refinement (i.e. it might be because the failure history changes the agent's decision even without the playbook). Without the original-playbook comparison and repeated adaptation runs, I am not convinced that these results establish reliable transfer.

Questions for authors
  • What happens when RobotUse and CaP-X use the same time or API-cost budget? Does the success advantage remain?

  • How much do visual candidate inspection and pose adjustment contribute? Please compare with a version using the same backend and context organization but automatically selecting the backend’s preferred grasp.

  • How does the system perform under substantial occlusion or incomplete depth observations? Can visual adjustment recover when the proposed grasp is geometrically feasible but physically unstable?

  • How does the original playbook perform on the same real-robot tasks? Does the refined playbook improve performance on tasks excluded from refinement, across repeated adaptation runs?

Comments on the paper

2 comments

Based on the current experimental results, RobotUse does not demonstrate a particularly strong success-rate advantage over existing agent-based robotic systems, nor does it offer a clear advantage over CaP-X in terms of computational cost. Many of the evaluated tasks are essentially pick-and-place tasks and do not appear particularly challenging or novel.

The authors identify the core contributions as enabling agents to actively adjust gripper positions, rotations, and end-effector poses in visual space; encapsulating frequent visual checks and pose adjustments within subgoal-local contexts while returning concise outcomes to the main agent; and allowing playbook experience to guide execution through an adjustable, error-correcting action interface. However, these appear to be fairly general design choices. Current models such as Astra and Opus already possess these capabilities without requiring a specialized harness. The paper does not provide evidence that the proposed framework can further improve or unlock the capabilities of these models. I therefore recommend evaluating the framework with Astra and Opus.

I appreciate the authors’ additional experiments and will raise my score to 2 (Weak rejection). However, the additional evidence is insufficient to address the paper’s underlying methodological limitations. I also encourage the authors to evaluate RobotUse on RoboDojo to better demonstrate its capabilities.

Dear Reviewers and Area Chair,

Thank you for the constructive feedback and engagement throughout the discussion period. As the discussion concludes, we would like to provide a brief summary of the additional experiments and clarifications shared in response to the main points raised across the reviews.

1. Main concern: Performance under matched budgets (SnCU, gtfT, pwqs) The primary concern across reviews was whether RobotUse's gains hold under matched API cost, time, and token budgets. To test this cleanly, we removed all external tools and grasp proposers, leaving only RobotUse's core elements (rendered pose previews and local context isolation), and went head-to-head against RoboDojo's official Direct_EEF harness using the same frontier VLM (Gemini Carbon, 50 episodes per method across 10 tasks). RobotUse achieves substantially higher success under every budget cap: 46.0% vs. 6.0% at <= $0.75 (7.7x), 40.0% vs. 8.0% at <= 15 min (5.0x), and 44.0% vs. 10.0% at <= 1.5M tokens (4.4x). Even uncapped, RobotUse outperforms Direct_EEF (64.0% vs. 62.0%) while cutting API cost by 31.9% and runtime by 30.4%.

2. Novelty: Distinct from passive selection and hardware calibration (SnCU, gtfT, pwqs) Controlled experiments directly separate our method from existing approaches:

  • vs. PIVOT (SnCU, gtfT): On the same RoboLab setting, standalone PIVOT achieves 0.0% (0/40) while costing more (\0.88 vs. \0.48) and taking longer (2,192s vs. 1,775s) than RobotUse (47.5%, 19/40). Active 6-DoF pose authoring with visual previews succeeds where passive 2D marker sampling fails.
  • Pure strategy transfer (SnCU, pwqs, gtfT): On 20 unseen RoboLab tasks in simulation (where camera-to-robot calibration is 100% exact and fixed), continual playbook evolution raises zero-shot success from 25.0% (Initial) to 35.0% (Round 2) and 40.0% (Round 3) (+15.0%p).

3. Interface design is not trivial, even for frontier models (gtfT, SnCU) Reviewer gtfT questioned whether frontier models already possess these capabilities without a specialized harness. Our proposal-free RoboDojo results with Gemini Carbon show the opposite: without rendered pose previews and subgoal-local context isolation, even a frontier VLM collapses under practical budgets (6.0% at <= $0.75, 2.0% at <= 10 min) due to unencapsulated visual-checking loops and context bloat. Aligning active 6-DoF decisions in visual space while isolating local debugging resolves a core bottleneck in embodied agents, consistent with recent empirical studies on frontier robot harnesses.

Throughout the rebuttal, we have responded promptly to every review and follow-up question, sharing new experimental results and rollout videos on our anonymous project page as soon as they finished. We hope this summary assists in the final evaluation.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.