- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CodeActionBench, a benchmark including 25 simulated dual-arm manipulation tasks designed to test the general purpose multimodal models if they can auto generate, execute, and iteratively refine executable code (Code-as-Policy) using only RGB observations, camera calibration, robot feedback, and a limited set of geometry tools to accomplish grasping, transport, handover, tool use, stacking, and articulated-object manipulation tasks without task fine-tuning, demonstrations, expert perception/grasping modules, privileged scene state, or predefined task policies.
A shared robot API and a hidden (out-of-band) verifier keep all task-relevant decisions on the evaluated agent's side, while physical outcomes are scored independently by the verifier.
The benchmark is evaluated on nine "model + harness" configurations over 675 attempts, with success rates ranging from 2.7% to 73.3%; GPT-6 Astra (Codex CLI) is the strongest, solving 22 of 25 tasks. The paper also contributes subgoal/spatial-checkpoint analysis(Checkpoint definitions), failure-stage decomposition(Failure analysis), and completion-claim (overclaim/underclaim) assessment methods(Task Completion Assessment and Termination).
Critical strengths
The research question is valuable and the evaluation protocol is documented in detail (shared API, fixed budgets, hidden verifier, full trajectory recording), with appendices that are more transparent than the average for this type of paper.
Critical weaknesses
The current evidence supports "performance on fixed instances" more strongly than the generalization ability implied by the paper's capability framing. Structural issues must be resolved before the conclusions can be trusted.
- The two variables model and harness are change together in experiment. The protocol section says that cross-harness comparisons evaluate "complete agent systems", yet the results section says the advantage is from the model.
- No statistical uncertainty or paired comparisons. Three randomized attempts in a single fixed scene can not report percentage-point estimates and a "model+harness" ranking. There are no confidence intervals, no paired tests, and no ranking-stability analysis anywhere, and the paper itself concedes that the number of attempts is insufficient.
- Reference positions and tolerances of the subgoal and spatial checkpoints come from successful expert executions and task geometry. I need to know the FN rate of unmatched checkpoints among successful attempts. Pls try to use independent reference trajectories, and perform a tolerance sensitivity analysis.
- There is a obvious gap between the Claude Opus 5+Reference configuration and the Claude Opus 5+Claude Code configuration. Test results should be added for the same model under different harnesses, to verify whether different harness configurations affect the model.
- Failure-stage labels (such as "no confirmed arrival") indicate the earliest stage that can be supported, not a causal diagnosis of perception or planning failure. It should be used consistently throughout and made explicit in the conclusions.
- Abstract opens with a capability question, but actually measures fixed instances rather than generalization across different instances.
- Pls clarify the novelty to the CaP-X 2026, distinguish the contemporaneous Agent as Policy 2026.
- Cost is an API-equivalent estimate at fixed rates. The paper should clearly distinguish "API-equivalent cost estimate" from "actual deployment cost," and "wall-clock time (including waiting)" from "algorithmic execution efficiency."
- At least one class of non-privileged baseline should be added, along with a per-task difficulty distribution and relative gains.
- The paper lacks of VLA / conventional-policy comparison baselines.
- The paper lacks of ablation of interface tools.
- Context-management variable is uncontrolled..
- Physical realism and safety of the collision model need to be recosidered. A control experiment with scene collision enabled need to be added.
- Over-operation (continuing to act after achieving the goal) is not separately quantified
- The paper also lacks of sensitivity analysis of physical parameters, and cross-scene evaluation.
Questions for authors
None.
9 comments
W3. Checkpoint false negatives, independent references and tolerance sensitivity
We thank the reviewer for this careful question. Checkpoint tolerances come from expert trajectories, and we validated them on the actual successful attempts: across the 522 successful attempts in three scenes, 8.2% of the spatial checkpoints they pass through are missed at the paper's tolerances, and 4.8% at 1.25× tolerance. Tolerances are those of the paper. Reference points are anchored to the task objects at the median of RoboTwin expert successes, 18–20 trajectories per task (487 in total), so no model attempt was used to set them. Checkpoints are offline diagnostics and do not affect success.
We also performed a tolerance sensitivity analysis, scaling every tolerance from 0.5× to 2×.
Spatial checkpoints missed in successful attempts, by tolerance multiplier.
| Successful attempts | ×0.5 | ×0.75 | ×1 | ×1.25 | ×1.5 | ×2 |
|---|---|---|---|---|---|---|
| Nine configurations (1,344 checkpoints) | 497 (37.0%) | 223 (16.6%) | 110 (8.2%) | 64 (4.8%) | 38 (2.8%) | 29 (2.2%) |
The largest share (20 of 110) comes from place_bread_basket, where agents dropped the bread 4–11 cm above the experts' release point, outside the −2 to +3 cm window; we keep this window, which reflects reliable placement. We will add these results to Appendix E.
W4. Opus 5 under two harnesses
We thank the reviewer for this observation, and we would like to clarify two points. First, Opus 5 shows no clear gap between the two harnesses. In the original scene, its attempts progress equally far under both harnesses: spatial checkpoint coverage is 73.4% under Claude Code and 73.6% under the reference harness (Figure 5). With the two new scenes added (W2), its success rate over the three scenes is 44.0% (99/225) under Claude Code and 42.2% (95/225) under the reference harness, and its spatial checkpoint coverage is comparable as well, at 74.3% and 77.0%, respectively. Second, we evaluate agents rather than compare harness choices. Our configurations pair either a model with the reference harness or a leading model with its native harness, to assess how well advanced and widely used agents control robots through code.
W5. Failure-stage labels are not causal diagnoses
We agree, and this is how the paper defines them; we apologize for the confusion. Each failed attempt is assigned "to its earliest evidence-supported failure stage" (Section 4.3, Figure 6); labels are worded as unconfirmed events, refer to the unmet subgoal, and cases without ordered evidence remain unresolved (Appendix F.1). The phrase "difficulties in spatial alignment" in the abstract and introduction refers to our spatial checkpoints, i.e., whether the gripper or object reaches the relevant region. Reaching it depends jointly on selecting visual evidence, 3D estimation, target construction and motion execution, so we do not reduce it to a perception or planning failure. We kindly ask the reviewer to revisit these sections.
W6. Capability framing
We thank the reviewer and agree: the submission evaluated one fixed instance per task, as Section 3.5 and Appendix H state. We have now added two new scenes per task, with different object layouts, object models or colours, and table and wall textures, and the results carry over to the new instances (W2).
We would also like to clarify the capability that the abstract asks about. It is not the accuracy of a single component such as perception or grasp-pose proposal, but how far a model can carry a manipulation task through code. Each stage of a task requires several components to work together (perception, 3D estimation, target construction, and policy execution and revision). We therefore define a general process that applies to every task and, using the checkpoints of Section 4.3, record the stage at which each attempt ends:
- Relevant regions not reached. A subgoal was never completed, and the gripper did not reach the region where each task object must be grasped or pressed.
- Reached, subgoals incomplete. The gripper reached every such region, but a subgoal was never completed.
- Subgoals completed, final state failed. Every subgoal was completed at some point, but the verifier's conditions did not hold when the attempt ended.
- Success. The hidden verifier accepted the attempt.
Where attempts end (three scenes, 225 attempts per configuration; the four stages are exclusive and cover every attempt).
| Configuration | Relevant regions not reached | Reached, subgoals incomplete | Subgoals completed, final state failed | Success |
|---|---|---|---|---|
| GPT-6 Astra (Codex CLI) | 4.0% | 21.3% | 2.7% | 72.0% |
| Opus 5 (Claude Code) | 19.6% | 32.4% | 4.0% | 44.0% |
| Opus 5 (Reference) | 16.0% | 35.6% | 6.2% | 42.2% |
| GPT-5.6 Sol (Reference) | 45.3% | 32.9% | 1.8% | 20.0% |
| Gemini 3.6 Flash (Reference) | 50.7% | 32.9% | 1.3% | 15.1% |
| Kimi K3 (Reference) | 53.3% | 29.8% | 4.0% | 12.9% |
| Qwen 3.8 Max (Reference) | 50.7% | 32.0% | 4.9% | 12.4% |
| Grok 4.6 (Reference) | 64.0% | 26.2% | 0.4% | 9.3% |
| Sonnet 5 (Reference) | 72.9% | 21.3% | 1.8% | 4.0% |
| All (2,025 attempts) | 41.8% | 29.4% | 3.0% | 25.8% |
The same profile holds in every scene: pooled over configurations, the four shares are 39.1%, 30.5%, 3.4% and 27.0% in the original scene, 41.9%, 28.9%, 3.1% and 26.1% in scene A, and 44.4%, 28.7%, 2.5% and 24.3% in scene B.
We will replace "Fixed task instances" in the abstract with "Task instances shared by all configurations" and report the three-scene results there.
W7. Novelty relative to CaP-X and Agent as Policy
We thank the reviewer for pointing to CaP-X, a highly relevant reference. Our novelty relative to CaP-X is that task-dependent visual interpretation and policy construction remain with the evaluated model: every CaP-X tier supplies object locations and grasp proposals, from privileged scene state or from external specialist modules on RGB-D input, whereas CodeActionBench withholds auxiliary specialist perception and grasp outputs, privileged scene state and predefined task policies, and evaluates agents under this boundary with a hidden verifier and process-level checkpoints. A detailed comparison is shown in Table 1 of the paper.
We appreciate Agent as Policy (AGP) [1], a high-quality concurrent work that explores the frontier of general coding agents as policies on real robots. The table below places AGP in the format of our Table 1, with one added column for demonstration input.
Interface boundaries of AGP and CodeActionBench.
| Method | No depth sensing | No privileged scene state | No specialist outputs | Agent-constructed 3D & grasp | No demonstration input |
|---|---|---|---|---|---|
| AGP [1] | ✗ | ✓ | ✓ | ✓ | ✗* |
| Agentic Code-as-Policy (ours) | ✓ | ✓ | ✓ | ✓ | ✓ |
The two settings differ in sensing and task specification. CodeActionBench provides no depth and specifies every task only by a language instruction, whereas AGP provides depth from a wrist-mounted RealSense D405 camera and specifies 6 of its 8 task configurations with a video or image. These are four pair assembly and two towel folding configurations, given a human demonstration video, and three block construction configurations, given a goal image; only dice flipping and targeted throwing use language alone.
[1] Mengzhao Jia, Yang Lin, Xixin Zhang, Zhihan Zhang, Xiaobai Liu, and Meng Jiang. Agent as Policy for Robotic Manipulation. arXiv preprint arXiv:2609.12541v3, 2026.
Response to Reviewer kMz7
We thank the reviewers (AW2T - R1, Lyuu - R2, kMz7 - R3) for their feedback. We are pleased that they found our research question valuable and timely (R2, R3), and our evaluation interesting, realistic, skill-free and task-agnostic (R1, R2). It is also noted that we provide comprehensive diagnosis beyond success rates (R1, R2, R3), broad experiments and clear figures (R1), and a transparent protocol (R3). We address your comments below.
W1. Model vs. harness
We thank the reviewer for the careful reading, and we fully agree that the harness is an important factor in embodied manipulation. We would like to clarify that CodeActionBench evaluates complete agent systems, and we do not intend to attribute the results to the model alone. Each configuration is defined by its model and harness, and comparisons across harnesses "evaluate complete agent systems" (Section 3.5). The reference harness, shared by seven models, allows a controlled comparison among those models under one harness, without implying that the remaining configurations differ only in the model. For this reason, every figure and table reports results per complete configuration (model and harness). We will state this explicitly at the start of Section 4.
W2 / W15. Statistical uncertainty, ranking stability and cross-scene evaluation
We thank the reviewer for this important point, which led us to extend the evaluation. With two new scenes per task (2,025 attempts in total), results hold across scenes (W15): the configuration ranking is stable, and 25 of the 36 configuration pairs differ significantly after multiple-comparison correction.
Each task now has three scenes: the original scene and two new scenes, A and B, and every configuration runs three attempts per scene (9 attempts per task, 225 per configuration). Each new scene uses a new RoboTwin scene seed, which re-samples the object layout, and all 50 new scenes change the object appearance (via model swap or recolour) and the table and wall textures (from RoboTwin 2.0's unseen texture set); RoboTwin's other randomizations (clutter, lighting, table height) remain off, as in the original scenes. This also addresses the cross-scene evaluation requested in W15.
The ranking is stable across scenes: for any two scenes, at least 30 of the 36 configuration pairs keep the same order (Kendall's τ 0.70–0.89, exact p ≤ 0.0083). GPT-6 Astra stays at 69–75% in every scene, so the results show a degree of cross-scene generalization rather than reflecting a single fixed scene (W15). For paired comparisons, we use an exact test conditioned on the successes of the two configurations within each task and scene, with Holm correction over all 36 pairs. GPT-6 Astra exceeds every other configuration (p < 10⁻¹²). The two Opus 5 configurations cannot be distinguished (44.0% vs. 42.2%, p = 0.70 before correction), and both exceed all remaining configurations. Among the remaining six, neighbouring configurations are not separable. We will add these results to Section 4.1 and the appendix.