- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents RobotUse, a robot-agent harness that separates task-level decision-making, visual action specification, and low-level robot execution. The framework employs a main agent for task planning, subagents for visually selecting and refining spatial actions, and a robot backend for geometric computation, motion planning, and control. It further introduces subgoal-local contexts to reduce context overhead and a persistent playbook that is updated from execution experience.
Experiments on RoboLab show that RobotUse achieves a 45.0% task success rate, outperforming CaP-X by 6.67 percentage points. The authors also evaluate context efficiency, performance improvements through playbook refinement, and execution on three real-robot manipulation tasks.
Critical strengths
The paper addresses a practical problem in organizing robot-agent execution. The separation of task-level reasoning, local spatial decisions, and backend motion control is clearly presented. The context-handoff ablation also provides useful evidence that isolating subgoal-specific interaction histories can substantially reduce token consumption compared with a single-agent implementation.
Critical weaknesses
1. Limited methodological novelty and insufficient justification of the cost–performance trade-off.
The proposed framework largely combines existing concepts, including multi-agent delegation, visual action selection, context isolation, and persistent playbook refinement. While these components form a functional system, the paper does not convincingly establish what fundamentally new methodology or insight RobotUse contributes beyond their integration.
Moreover, the performance improvement over CaP-X is relatively modest (45.0% vs. 38.33%), despite requiring approximately three times the API cost and execution time, and nearly five times the token consumption. The compared methods also use different execution budgets and stopping criteria. Without comparisons under matched resource constraints, it is difficult to determine whether the additional complexity and computational cost of RobotUse are justified.
2. Insufficient evidence for generalizable learning and real-world effectiveness.
The continual harnessing experiments repeatedly refine and evaluate the playbook on the same 40 RoboLab tasks. Although the final evaluation uses new episodes, the task set remains unchanged. As a result, the reported improvement from 47.5% to 55.0% may primarily reflect adaptation to recurring task-specific failures rather than transferable knowledge.
The real-robot evaluation is also limited to three relatively simple manipulation tasks. Importantly, the authors refine the playbook until an initial success before conducting the subsequent evaluations. Therefore, the reported 100% success rates reflect performance after task-specific adaptation rather than zero-shot generalization. The adaptation cost is not sufficiently characterized, and the experiments provide limited evidence that RobotUse can reliably handle more complex, long-horizon manipulation tasks.
Questions for authors
Q1. Methodological novelty compared with existing agent harnesses.
The proposed framework appears to build primarily on established techniques such as subagent delegation, visual action selection, and persistent experience-based guidance. Could the authors clarify what distinguishes RobotUse from existing approaches such as PIVOT and Continual Harness? What specific methodological contribution or new insight does this framework provide beyond combining these components?
Q2. Fair comparison under matched computational budgets.
RobotUse achieves a relatively modest improvement over CaP-X while requiring substantially more API cost, tokens, and execution time. Additionally, the compared methods use different execution budgets and stopping criteria. Could the authors provide comparisons under matched execution time, API cost, or interaction budgets? This would help determine whether RobotUse offers a meaningful advantage beyond simply allocating more resources to each task.
Q3. Generalization of continual playbook refinement.
The playbook is repeatedly refined and evaluated on the same 40 RoboLab tasks, making it difficult to distinguish generalizable learning from adaptation to task-specific failure patterns. Could the authors evaluate the refined playbook on a held-out set of tasks that were never used during refinement? Such an experiment would provide stronger evidence that RobotUse learns transferable strategies rather than becoming increasingly specialized to the evaluation task catalog.
Q4. Real-robot evaluation and adaptation cost.
The real-robot experiments cover only three basic manipulation tasks, and the playbook is refined until an initial success before the reported evaluations. Could the authors clarify the number of attempts, execution time, and computational cost required during this adaptation process? It would also be valuable to compare methods under equivalent adaptation opportunities and evaluate RobotUse on more challenging, multi-stage manipulation tasks. This would help establish whether the proposed harness provides practical benefits beyond reliable repetition of relatively simple, previously adapted tasks.
Based on the current experimental results, RobotUse does not demonstrate a particularly strong success-rate advantage over existing agent-based robotic systems, nor does it offer a clear advantage over CaP-X in terms of computational cost. Many of the evaluated tasks are essentially pick-and-place tasks and do not appear particularly challenging or novel.
The authors identify the core contributions as enabling agents to actively adjust gripper positions, rotations, and end-effector poses in visual space; encapsulating frequent visual checks and pose adjustments within subgoal-local contexts while returning concise outcomes to the main agent; and allowing playbook experience to guide execution through an adjustable, error-correcting action interface. However, these appear to be fairly general design choices. Current models such as Astra and Opus already possess these capabilities without requiring a specialized harness. The paper does not provide evidence that the proposed framework can further improve or unlock the capabilities of these models. I therefore recommend evaluating the framework with Astra and Opus.
I appreciate the authors’ additional experiments and will raise my score to 2 (Weak rejection). However, the additional evidence is insufficient to address the paper’s underlying methodological limitations. I also encourage the authors to evaluate RobotUse on RoboDojo to better demonstrate its capabilities.