- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes Agent as Policy (AGP), which uses a general-purpose coding agent directly as a policy for robotic manipulation tasks. The agent requests observations and actions using a command-line interface and receives proprioceptive and visual observations as well as action outcomes, writing programs during execution to process observations and plan actions, and adapting based on motion feedback and observations from failed attempts. To improve efficiency, the paper proposes experience refinement, wherein an agent is informed by recorded experience that it maintains over multiple rollouts of the same task, and experience transfer, wherein relevant experience recorded by a stronger agent performing a given task is used to inform a weaker model performing the same task.
The approach is validated on five different manipulation tasks testing capabilities in long-horizon reasoning, action diversity and object diversity. The paper also compares performance across agent models on an assembly task, compares AGP against two baseline methods on three tasks, tests experience refinement and transfer on two tasks and one task respectively, and provides a case study on failure recovery. The authors report substantially higher success rates than the baselines and observe that past experience can facilitate more efficient task completion and can be transferred across agent models.
Critical strengths
- AGP shows that an off-the-shelf coding agent supplied with only a thin robot interface can complete difficult real-world manipulation tasks zero-shot, including video-guided assembly with sub-millimetre clearances and bimanual towel folding, and succeeds in 23/25 trials against 3/25 and 0/25 for the baselines (Table 3). This is a non-obvious capability result and the main reason the paper merits consideration despite its thin methodological contribution.
Critical weaknesses
-
AGP's per-rollout cost and latency limit its practicality for repeated deployment, and the paper's efficiency comparison with ASPIRE obscures this since Section 4.2.3 applies ASPIRE's full preparation cost to every successful run.
- Each AGP rollout incurs the (significant) full inference cost (mean 7.5--24.1 USD and 17.7--50.8 minutes per rollout; Table 1). In contrast, the programs generated by ASPIRE replay with no further inference cost, amortizing the high initial cost over many rollouts. Indeed, from the results for two-pair assembly in Table 3 (ASPIRE's fixed USD 18.90 vs AGP's USD 4.47 per rollout), I would expect ASPIRE to become cheaper per success after about 11 rollouts (assuming ASPIRE maintains its observed 40% success rate). Given that robots would often be expected to complete the same tasks many times over, this seems like a major limitation regarding the scalability of AGP, and a more representative comparison would consider accumulated cost over many rollouts.
- Experience refinement and transfer (Section 4.3) reduce the per-rollout cost but not its linear scaling. Refinement reports only time (not token cost), and the ring task's 53% reduction mainly reflects a failed insertion in the first cycle (Appendix D.2). Transfer lets the cheaper Terra model execute, but each Terra rollout still costs roughly USD 1.75 (5.56M tokens at the rate implied by Table 2) and takes 12.5 minutes against ASPIRE's 2.9; assuming Terra's observed 80% success rate, I would expect ASPIRE to become cheaper per success after about 22 rollouts.
- The agent's decision loop operates on a timescale of minutes (e.g., the recovery steps in Figure 3a each span 1--4 minutes), and throwing relies on a task-specific runtime that executes pre-authored timed programs (Appendix A.1), so the paper's claim of "dynamic motions" is overstated.
- None of this is discussed as a limitation; the Broader Impact section (Appendix G) instead proposes AGP for collecting VLA training trajectories, which these costs make hard to justify compared to teleoperation.
-
No ablation isolates the central claim that using the agent directly as the runtime policy is what drives performance, and several comparative and statistical claims go beyond what the evaluation supports. This is particularly problematic since the methodological contribution is thin: AGP consists of an off-the-shelf agent, a seven-command robot interface and task prompts (Section 3, Appendix A.1), with experience reuse amounting to saved notes and scripts (Section 4.3). While simplicity is not itself a weakness, the paper's value therefore rests on its empirical claims.
- A key missing ablation is the same agent and interface generating a single program up-front without runtime feedback, in the style of Code as Policies (Liang et al., 2022), which needs no simulator. Without it, gains cannot be attributed to runtime agency rather than the model's ability to write perception code (29 of 37 retained programs; Appendix E), so the findings say little beyond the specific model snapshot.
- Baseline failures are attributed to the sim-to-real gap without reporting the baselines' success in simulation. GaP's 0/25 success also raises concerns about how faithfully it was reproduced.
- The evaluation is sparse relative to the claims drawn from it. The agent comparison uses one task (Table 2), extended to a second task for only two models (Appendix C), baselines are compared only on three tasks the authors describe as "relatively easy" (Section 4.2.3), refinement uses a single five-run sequence on each of two tasks, and transfer is tested on one task and one model pair, with no confidence intervals reported. Despite this, Section 4.3 states that the refinement and transfer results "prove" their effects, though transfer compares 1/5 with 4/5 successes (by applying a two-sided Fisher's exact test I arrived at p$\approx$0.21), with time and token reductions computed from one run against four (Appendix D.3).
- Protocol details are missing or weak. Budgets are never stated, the trial-inclusion policy is undescribed despite documented supervisor interventions (Appendices B.2 and C), and some successes are self-reported or leniently judged (Appendices D.2, I.5).
- Appendix F's error analysis is never referenced in the main text, including a case where the agent incorrectly reports an incomplete fold as complete (Appendix F.5); this is especially relevant because AGP itself decides when a task is finished (Section 3.3), so such verification errors would go undetected without a human assessor.
- All quantitative experiments use one tabletop setup with trials starting from predefined scenes (Section 4.1.2), so the claim of zero-shot execution in "previously unseen environments" (Sections 2.1 and 4.2.3) is tested only by a single qualitative demonstration on a second platform (Appendix H).
Questions for authors
- Amortized cost (clarification): How do AGP's and ASPIRE's cumulative cost and time compare as a function of the number of rollouts, including failed trials? The current comparison applies ASPIRE's full preparation cost to each success.
- Baseline validity (clarification): What are GaP's and ASPIRE's success rates in simulation? This would clarify whether the baselines' failures really reflect the sim-to-real gap.