acceptodds
Withdrawn from ICLR 2027

CodeActionBench: Evaluating Agentic Code-as-Policy for Embodied Manipulation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

How well can general-purpose multimodal models turn visual understanding and reasoning into embodied manipulation via executable code? We introduce CodeActionBench, a benchmark of 25 manipulation tasks that evaluates this capability through agentic Code-as-Policy. Without task-specific fine-tuning, demonstrations, external specialist perception or grasp modules, privileged scene state, or predefined task policies, agents should select visual evidence, form task-relevant 3D estimates, construct manipulation targets, and iteratively execute and revise their policies. A shared robot API provides RGB observations, calibrated geometric operations, robot feedback, and bounded motion, leaving task-dependent decisions to the evaluated agent. Fixed task instances, resource budgets, and a hidden physical-outcome verifier support controlled comparisons across models and harness configurations. Extensive evaluations across nine configurations and 675 attempts achieve success rates ranging from 2.7% to 73.3%. The strongest configuration, GPT-6 Astra with Codex CLI, solves 22 of 25 tasks at least once in three attempts, demonstrating the best performance while still leaving substantial room for improvement. Trajectory analyses reveal difficulties in spatial alignment, object retention, and completion judgment, including task failures despite successfully completed motions. CodeActionBench provides a controlled testbed for measuring how general-purpose models translate their capabilities into manipulation behavior and for examining typical failure scenarios in that process.

Primary area:
datasets and benchmarks
Keywords:
embodied AI, robot manipulation, code-as-policy, LLM agent, benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_kMz7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CodeActionBench, a benchmark including 25 simulated dual-arm manipulation tasks designed to test the general purpose multimodal models if they can auto generate, execute, and iteratively refine executable code (Code-as-Policy) using only RGB observations, camera calibration, robot feedback, and a limited set of geometry tools to accomplish grasping, transport, handover, tool use, stacking, and articulated-object manipulation tasks without task fine-tuning, demonstrations, expert perception/grasping modules, privileged scene state, or predefined task policies.

A shared robot API and a hidden (out-of-band) verifier keep all task-relevant decisions on the evaluated agent's side, while physical outcomes are scored independently by the verifier.

The benchmark is evaluated on nine "model + harness" configurations over 675 attempts, with success rates ranging from 2.7% to 73.3%; GPT-6 Astra (Codex CLI) is the strongest, solving 22 of 25 tasks. The paper also contributes subgoal/spatial-checkpoint analysis(Checkpoint definitions), failure-stage decomposition(Failure analysis), and completion-claim (overclaim/underclaim) assessment methods(Task Completion Assessment and Termination).

Critical strengths

The research question is valuable and the evaluation protocol is documented in detail (shared API, fixed budgets, hidden verifier, full trajectory recording), with appendices that are more transparent than the average for this type of paper.

Critical weaknesses

The current evidence supports "performance on fixed instances" more strongly than the generalization ability implied by the paper's capability framing. Structural issues must be resolved before the conclusions can be trusted.

  1. The two variables model and harness are change together in experiment. The protocol section says that cross-harness comparisons evaluate "complete agent systems", yet the results section says the advantage is from the model.
  2. No statistical uncertainty or paired comparisons. Three randomized attempts in a single fixed scene can not report percentage-point estimates and a "model+harness" ranking. There are no confidence intervals, no paired tests, and no ranking-stability analysis anywhere, and the paper itself concedes that the number of attempts is insufficient.
  3. Reference positions and tolerances of the subgoal and spatial checkpoints come from successful expert executions and task geometry. I need to know the FN rate of unmatched checkpoints among successful attempts. Pls try to use independent reference trajectories, and perform a tolerance sensitivity analysis.
  4. There is a obvious gap between the Claude Opus 5+Reference configuration and the Claude Opus 5+Claude Code configuration. Test results should be added for the same model under different harnesses, to verify whether different harness configurations affect the model.
  5. Failure-stage labels (such as "no confirmed arrival") indicate the earliest stage that can be supported, not a causal diagnosis of perception or planning failure. It should be used consistently throughout and made explicit in the conclusions.
  6. Abstract opens with a capability question, but actually measures fixed instances rather than generalization across different instances.
  7. Pls clarify the novelty to the CaP-X 2026, distinguish the contemporaneous Agent as Policy 2026.
  8. Cost is an API-equivalent estimate at fixed rates. The paper should clearly distinguish "API-equivalent cost estimate" from "actual deployment cost," and "wall-clock time (including waiting)" from "algorithmic execution efficiency."
  9. At least one class of non-privileged baseline should be added, along with a per-task difficulty distribution and relative gains.
  10. The paper lacks of VLA / conventional-policy comparison baselines.
  11. The paper lacks of ablation of interface tools.
  12. Context-management variable is uncontrolled..
  13. Physical realism and safety of the collision model need to be recosidered. A control experiment with scene collision enabled need to be added.
  14. Over-operation (continuing to act after achieving the goal) is not separately quantified
  15. The paper also lacks of sensitivity analysis of physical parameters, and cross-scene evaluation.
Questions for authors

None.

9 comments

Response to Reviewer kMz7

We thank the reviewers (AW2T - R1, Lyuu - R2, kMz7 - R3) for their feedback. We are pleased that they found our research question valuable and timely (R2, R3), and our evaluation interesting, realistic, skill-free and task-agnostic (R1, R2). It is also noted that we provide comprehensive diagnosis beyond success rates (R1, R2, R3), broad experiments and clear figures (R1), and a transparent protocol (R3). We address your comments below.

 


 

W1. Model vs. harness

We thank the reviewer for the careful reading, and we fully agree that the harness is an important factor in embodied manipulation. We would like to clarify that CodeActionBench evaluates complete agent systems, and we do not intend to attribute the results to the model alone. Each configuration is defined by its model and harness, and comparisons across harnesses "evaluate complete agent systems" (Section 3.5). The reference harness, shared by seven models, allows a controlled comparison among those models under one harness, without implying that the remaining configurations differ only in the model. For this reason, every figure and table reports results per complete configuration (model and harness). We will state this explicitly at the start of Section 4.

 

W2 / W15. Statistical uncertainty, ranking stability and cross-scene evaluation

We thank the reviewer for this important point, which led us to extend the evaluation. With two new scenes per task (2,025 attempts in total), results hold across scenes (W15): the configuration ranking is stable, and 25 of the 36 configuration pairs differ significantly after multiple-comparison correction.

Each task now has three scenes: the original scene and two new scenes, A and B, and every configuration runs three attempts per scene (9 attempts per task, 225 per configuration). Each new scene uses a new RoboTwin scene seed, which re-samples the object layout, and all 50 new scenes change the object appearance (via model swap or recolour) and the table and wall textures (from RoboTwin 2.0's unseen texture set); RoboTwin's other randomizations (clutter, lighting, table height) remain off, as in the original scenes. This also addresses the cross-scene evaluation requested in W15.

ConfigurationOriginal sceneNew scene ANew scene BAll three scenes
GPT-6 Astra (Codex CLI)54/7556/7552/75162/225 (72.0%)
Opus 5 (Claude Code)33/7536/7530/7599/225 (44.0%)
Opus 5 (Reference)37/7530/7528/7595/225 (42.2%)
GPT-5.6 Sol (Reference)12/7514/7519/7545/225 (20.0%)
Gemini 3.6 Flash (Reference)15/759/7510/7534/225 (15.1%)
Kimi K3 (Reference)9/7511/759/7529/225 (12.9%)
Qwen 3.8 Max (Reference)11/7510/757/7528/225 (12.4%)
Grok 4.6 (Reference)9/757/755/7521/225 (9.3%)
Sonnet 5 (Reference)2/753/754/759/225 (4.0%)
Total182/675176/675164/675522/2,025 (25.8%)

The ranking is stable across scenes: for any two scenes, at least 30 of the 36 configuration pairs keep the same order (Kendall's τ 0.70–0.89, exact p ≤ 0.0083). GPT-6 Astra stays at 69–75% in every scene, so the results show a degree of cross-scene generalization rather than reflecting a single fixed scene (W15). For paired comparisons, we use an exact test conditioned on the successes of the two configurations within each task and scene, with Holm correction over all 36 pairs. GPT-6 Astra exceeds every other configuration (p < 10⁻¹²). The two Opus 5 configurations cannot be distinguished (44.0% vs. 42.2%, p = 0.70 before correction), and both exceed all remaining configurations. Among the remaining six, neighbouring configurations are not separable. We will add these results to Section 4.1 and the appendix.

W3. Checkpoint false negatives, independent references and tolerance sensitivity

We thank the reviewer for this careful question. Checkpoint tolerances come from expert trajectories, and we validated them on the actual successful attempts: across the 522 successful attempts in three scenes, 8.2% of the spatial checkpoints they pass through are missed at the paper's tolerances, and 4.8% at 1.25× tolerance. Tolerances are those of the paper. Reference points are anchored to the task objects at the median of RoboTwin expert successes, 18–20 trajectories per task (487 in total), so no model attempt was used to set them. Checkpoints are offline diagnostics and do not affect success.

We also performed a tolerance sensitivity analysis, scaling every tolerance from 0.5× to 2×.

Spatial checkpoints missed in successful attempts, by tolerance multiplier.

Successful attempts×0.5×0.75×1×1.25×1.5×2
Nine configurations (1,344 checkpoints)497 (37.0%)223 (16.6%)110 (8.2%)64 (4.8%)38 (2.8%)29 (2.2%)

The largest share (20 of 110) comes from place_bread_basket, where agents dropped the bread 4–11 cm above the experts' release point, outside the −2 to +3 cm window; we keep this window, which reflects reliable placement. We will add these results to Appendix E.

 

W4. Opus 5 under two harnesses

We thank the reviewer for this observation, and we would like to clarify two points. First, Opus 5 shows no clear gap between the two harnesses. In the original scene, its attempts progress equally far under both harnesses: spatial checkpoint coverage is 73.4% under Claude Code and 73.6% under the reference harness (Figure 5). With the two new scenes added (W2), its success rate over the three scenes is 44.0% (99/225) under Claude Code and 42.2% (95/225) under the reference harness, and its spatial checkpoint coverage is comparable as well, at 74.3% and 77.0%, respectively. Second, we evaluate agents rather than compare harness choices. Our configurations pair either a model with the reference harness or a leading model with its native harness, to assess how well advanced and widely used agents control robots through code.

 

W5. Failure-stage labels are not causal diagnoses

We agree, and this is how the paper defines them; we apologize for the confusion. Each failed attempt is assigned "to its earliest evidence-supported failure stage" (Section 4.3, Figure 6); labels are worded as unconfirmed events, refer to the unmet subgoal, and cases without ordered evidence remain unresolved (Appendix F.1). The phrase "difficulties in spatial alignment" in the abstract and introduction refers to our spatial checkpoints, i.e., whether the gripper or object reaches the relevant region. Reaching it depends jointly on selecting visual evidence, 3D estimation, target construction and motion execution, so we do not reduce it to a perception or planning failure. We kindly ask the reviewer to revisit these sections.

W6. Capability framing

We thank the reviewer and agree: the submission evaluated one fixed instance per task, as Section 3.5 and Appendix H state. We have now added two new scenes per task, with different object layouts, object models or colours, and table and wall textures, and the results carry over to the new instances (W2).

We would also like to clarify the capability that the abstract asks about. It is not the accuracy of a single component such as perception or grasp-pose proposal, but how far a model can carry a manipulation task through code. Each stage of a task requires several components to work together (perception, 3D estimation, target construction, and policy execution and revision). We therefore define a general process that applies to every task and, using the checkpoints of Section 4.3, record the stage at which each attempt ends:

  • Relevant regions not reached. A subgoal was never completed, and the gripper did not reach the region where each task object must be grasped or pressed.
  • Reached, subgoals incomplete. The gripper reached every such region, but a subgoal was never completed.
  • Subgoals completed, final state failed. Every subgoal was completed at some point, but the verifier's conditions did not hold when the attempt ended.
  • Success. The hidden verifier accepted the attempt.

Where attempts end (three scenes, 225 attempts per configuration; the four stages are exclusive and cover every attempt).

ConfigurationRelevant regions not reachedReached, subgoals incompleteSubgoals completed, final state failedSuccess
GPT-6 Astra (Codex CLI)4.0%21.3%2.7%72.0%
Opus 5 (Claude Code)19.6%32.4%4.0%44.0%
Opus 5 (Reference)16.0%35.6%6.2%42.2%
GPT-5.6 Sol (Reference)45.3%32.9%1.8%20.0%
Gemini 3.6 Flash (Reference)50.7%32.9%1.3%15.1%
Kimi K3 (Reference)53.3%29.8%4.0%12.9%
Qwen 3.8 Max (Reference)50.7%32.0%4.9%12.4%
Grok 4.6 (Reference)64.0%26.2%0.4%9.3%
Sonnet 5 (Reference)72.9%21.3%1.8%4.0%
All (2,025 attempts)41.8%29.4%3.0%25.8%

The same profile holds in every scene: pooled over configurations, the four shares are 39.1%, 30.5%, 3.4% and 27.0% in the original scene, 41.9%, 28.9%, 3.1% and 26.1% in scene A, and 44.4%, 28.7%, 2.5% and 24.3% in scene B.

We will replace "Fixed task instances" in the abstract with "Task instances shared by all configurations" and report the three-scene results there.

 

W7. Novelty relative to CaP-X and Agent as Policy

We thank the reviewer for pointing to CaP-X, a highly relevant reference. Our novelty relative to CaP-X is that task-dependent visual interpretation and policy construction remain with the evaluated model: every CaP-X tier supplies object locations and grasp proposals, from privileged scene state or from external specialist modules on RGB-D input, whereas CodeActionBench withholds auxiliary specialist perception and grasp outputs, privileged scene state and predefined task policies, and evaluates agents under this boundary with a hidden verifier and process-level checkpoints. A detailed comparison is shown in Table 1 of the paper.

We appreciate Agent as Policy (AGP) [1], a high-quality concurrent work that explores the frontier of general coding agents as policies on real robots. The table below places AGP in the format of our Table 1, with one added column for demonstration input.

Interface boundaries of AGP and CodeActionBench.

MethodNo depth sensingNo privileged scene stateNo specialist outputsAgent-constructed 3D & graspNo demonstration input
AGP [1]✗✓✓✓✗*
Agentic Code-as-Policy (ours)✓✓✓✓✓

The two settings differ in sensing and task specification. CodeActionBench provides no depth and specifies every task only by a language instruction, whereas AGP provides depth from a wrist-mounted RealSense D405 camera and specifies 6 of its 8 task configurations with a video or image. These are four pair assembly and two towel folding configurations, given a human demonstration video, and three block construction configurations, given a goal image; only dice flipping and targeted throwing use language alone.

[1] Mengzhao Jia, Yang Lin, Xixin Zhang, Zhihan Zhang, Xiaobai Liu, and Meng Jiang. Agent as Policy for Robotic Manipulation. arXiv preprint arXiv:2609.12541v3, 2026.

Reviewer_LyuuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CodeActionBench, which evaluates whether general-purpose multimodal models can combine visual understanding, spatial estimation, and policy construction through executable code to perform robotic manipulation. Built on SAPIEN and RoboTwin 2.0, the benchmark includes 25 tasks with fixed scene instances. Models receive RGB images, camera calibration, robot state, and anonymous contact feedback. They use a shared API for geometric calculations and motion control, select corresponding image points, construct grasp targets, and revise their policies based on execution feedback. The evaluation covers nine model configurations, each making three attempts on each of the 25 tasks, for 75 attempts per configuration. Success rates range from 2.7% to 73.3%. The best-performing configuration, GPT-6 Astra with Codex CLI, succeeds in 55 of 75 attempts and solves 22 of the 25 tasks at least once across three attempts. CodeActionBench provides a controlled testbed for measuring how general-purpose models translate their capabilities into manipulation behavior and for examining common failures along the way. Overall, it explores an alternative to current end-to-end VLA and WAM policies: a "brain + cerebellum" control setup that connects high-level reasoning with low-level control. I think this setting fits well with the field's current direction.

Critical strengths
  1. The evaluation setup makes sense. Models are given neither depth information nor ground-truth scene state, which seems closer to real-world inference conditions. The "brain + cerebellum" control setup also offers an alternative to current end-to-end VLA and WAM policies. I think this is a timely direction to explore.

  2. The framework supports detailed diagnosis. It goes beyond final success rates and examines what happens during execution, helping identify how models perform at different stages and where they struggle. The failure analysis is fairly comprehensive.

Critical weaknesses
  1. Some tasks require specific steps, but it is unclear whether the scoring checks those steps. For example, the block handover task asks the robot to pick up the block with the left hand, pass it to the right hand, and then place it on the pad (Table 3, p. 17). However, the paper says that these tasks are scored based on the final state (Section 3.4 and Appendix A.3). Seeing the block on the pad at the end does not tell us whether a handover actually happened. If placing the block directly on the pad also counts as success, that result cannot demonstrate the model's ability to perform a bimanual handover. The authors should explain how the scoring checks these steps. At this point, the explanation is incomplete, but that does not establish that any results were actually scored incorrectly.

  2. The rules for including runs in the results and stopping attempts early are not fully explained. The paper says that some superseded runs and infrastructure failures were excluded, but does not report how many runs were excluded for each model or how replacement runs were selected (Section 3.5, p. 6). In addition, eight Kimi attempts ended because they took too long in wall-clock time, but the paper does not give this time limit or explain whether it applied equally to all models (Appendix G.2 and Table 15, pp. 31–32). These rules can affect success rates and need to be clear. If all models followed the same rules set in advance, a clarification would be enough. If the rules differed, the comparison might not be fair.

Questions for authors
  1. What do the blue arrow and the label "A36.9 Deg" mean in the Pose preview panel of Figure 2? I found this confusing and could not find an explanation in the paper.

  2. Does the scoring for handover tasks actually check that a handover took place? For example, the task asks the left hand to pass the block to the right hand before placing it on the pad. If the robot skips the handover and simply places the block on the pad, would it still be counted as a success? Please explain how the scoring checks this and whether the trajectories marked as successful completed the required steps (Table 3, pp. 16–17). If these steps are already checked, that would resolve my concern. If not, the scores would not be enough to demonstrate handover ability.

  3. Did all models follow the same rules when runs were excluded, replaced, or stopped early? Please report how many attempts were excluded for each model, why they were excluded, and how replacement attempts were selected. Also, eight Kimi attempts were stopped because they took too long. What was the time limit, and was it the same for all models (Section 3.5 and Table 15)? If these rules were set in advance and applied consistently, a clarification would be enough. If not, the fairness of the success-rate comparison would need to be reconsidered.

1 comment
AuthorsOpenReview ↗

Thank you for your constructive feedback!

We thank the reviewers (AW2T - R1, Lyuu - R2, kMz7 - R3) for their feedback. We are pleased that they found our research question valuable and timely (R2, R3), and our evaluation interesting, realistic, skill-free and task-agnostic (R1, R2). It is also noted that we provide comprehensive diagnosis beyond success rates (R1, R2, R3), broad experiments and clear figures (R1), and a transparent protocol (R3). We address your comments below.

 


 

W1 / Q2. Does the scoring check that a handover took place?

We thank the reviewer for this careful question. At submission, our evaluation included transfer semantics but no strict hand-to-hand constraint: the spatial checkpoint credited either gripper for reaching a handover region, so transfers via the table also passed. We have therefore added a direct-handover requirement to both the checkpoint and the verifier: the giving hand must hold the object alone and pass it to the receiving hand, which must then hold it alone, without the object returning to the table in between. Success now requires this handover and the original final state.

Following the other reviewers' comments, we added two new scenes to each task (new object layouts, instances and textures; 1,350 further attempts) to validate our results across scenes. Across all three scenes, every attempt that reached the final state but skipped the handover had set the object down on the table for the other hand to pick up, and all three are now scored as failures.

Task (three scenes)AttemptsFinal state reachedCompleted the required steps (success)Skipped the handover (now failure)
handover_block811192
handover_mic8120191
Total16231283

The revised rule is in §3.4, Appendix A.3 and Table 3, and the contact criterion in Appendix E.3.

 

W2 / Q3. Exclusions and wall-clock limits across models

We thank the reviewer for stressing that these rules must be clear for a valid comparison. We want to clarify that the same rules for excluded attempts and wall-clock limits applied in all three scenes (the original scene and the two new scenes described in W1), and the tables below report both.

Attempts stopped early by infrastructure errors (§3.5 and Appendix G.2).

All models followed the same rules when attempts were excluded, replaced, or stopped early. An attempt stopped by a provider, harness or simulator error was excluded, and the next valid attempt of the same cell counted once, whether it succeeded or failed.

ConfigurationOriginal sceneTwo new scenesTotal
GPT-6 Astra (Codex CLI)011
Opus 5 (Claude Code)000
Opus 5 (Reference)088
GPT-5.6 Sol (Reference)088
Gemini 3.6 Flash (Reference)066
Kimi K3 (Reference)2810
Qwen 3.8 Max (Reference)11314
Grok 4.6 (Reference)145
Sonnet 5 (Reference)022
Total45054

Wall-clock timeouts (Table 15).

For each task, all nine configurations shared the same wall-clock limit, set before evaluation to one minute per budgeted tool call (Table 3). Attempts that reached it counted as failures, and all of them exceeded their task limits even after we subtracted infrastructure waits. In the original scene, only Kimi reached the limit.

 

Q1. Blue arrow and angle label in Figure 2

We thank the reviewer for pointing this out. The agent specifies a gripper pose in preview_tcp_pose and sees it overlaid on the selected RGB image. The blue arrow shows the image projection of the gripper’s wrist-to-fingertip axis. Its label (36.9° in Figure 2) gives the unsigned 3D angle between that axis and the camera’s optical axis, so the agent can tell whether the gripper axis lies closer to the line of sight or to the image plane. Appendix Figure 7 uses the same overlay. We have revised its caption to explain the blue arrow and its angle label.

 


 

We hope these corrections and clarifications address your concerns, and we would appreciate it if you would consider raising your score.

Reviewer_AW2TOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a new benchmark for agentic robotics called CodeActionBench, which is an assortment of 25 simulated tasks along with a harness which allows an LLM agent to interface with the robot via a low-level observation and action API. In contrast to more standard code-as-policy scenarios where the LLM agent is provided a set of tools to interact with the environment which includes perception and action primitives, in CodeActionBench agents fully interact through a simpler API where they must design perception and action tools themselves based on visual feedback. The authors evaluate several LLMs in this domain and find a wide range of performance between frontier coding models. Experiments are evaluated on each of the 25 tasks with 3 trials each per LLM.

Critical strengths
  • The figures are clear as to the difference between CodeActionBench evaluation and prior evaluation methodologies.
  • The evaluation framework is an interesting way of evaluating a more “raw” performance metric on VLM capabilities with less explicit input from the designers of the harness.
  • Experiments cover a wide range of LLMs, and failure cases or behaviors (for example subgoal success and task coverage).
Critical weaknesses
  • The primary contribution of these 25 tasks is unclear compared to prior work. The introduction states they introduce a benchmark “in which task-dependent visual interpretation and policy construction remain with the evaluated model”. In my interpretation, that means that the LLM being evaluated must write the tool-calls / skills itself for the various perception and manipulation tasks being done. This seems less like a benchmark-level design decision and more like a specific decision on how to implement a CaP agent, irrespective of tasks or benchmark. Perhaps the contribution would be better framed as testing the ability of coding agents to code their own skills in a closed loop, rather than suggesting an entirely new benchmark? In this way previous benchmarks could also be pooled to provide a wider range of tasks to evaluate on.

  • One of the most interesting aspects of the paper is evaluating coding agents for robotics without explicit pre-constructed skills available. However, the paper is missing an experiment isolating the sensitivity of models to this design choice on the same tasks in CodeActionBench. Scientifically, this seems like one of the most important experiments to include because it directly answers the question of how sensitive coding agents are to the detailed skills provided in the Code-as-Policy style of evaluation. As is, the paper evaluates models only in the purely agentic framework without comparison to a harness which provides explicit skills (i.e. the methods in Table 1, and Figure 1a). Without this context it’s difficult to assess the performance difference between the authors’ proposed method of running agents and the standard CaP APIs.

  • While there are many raw results, I wish there was more synthesis of them into broader takeaways about agentic robotics, benchmarks, or harness design. How does the proposed benchmark and results influence what future directions should focus on?

  • The majority of tasks are very simple pick and place (11 tasks), and there seems to be quite a lot of overlap between tasks such as lifting different objects vs pick + place.

  • In addition, each scene is only evaluated on 1 configuration of objects, with 3 trials on the same configuration. Given that evaluation is in simulation I would have expected a significantly higher amount of variation within each task (ie different object positions and variations)

Questions for authors
  • Could the authors provide experiments comparing CodeActionBench tasks with agentic code actions vs the more canonical code-as-policy harness with robot skills provided?
  • Similarly can the authors provide experiments with more configurations of objects per task? that would make the benchmark much more compelling.
3 comments

Response to Reviewer AW2T

We thank the reviewers (AW2T - R1, Lyuu - R2, kMz7 - R3) for their feedback. We are pleased that they found our research question valuable and timely (R2, R3), and our evaluation interesting, realistic, skill-free and task-agnostic (R1, R2). It is also noted that we provide comprehensive diagnosis beyond success rates (R1, R2, R3), broad experiments and clear figures (R1), and a transparent protocol (R3). We address your comments below.

 


 

W1. Contribution as a benchmark

We thank the reviewer for this perspective, and we agree that the core of our setting is coding agents building their own skills in a closed loop. CodeActionBench aims to provide a common ground for studying this setting: a controlled testbed that evaluates manipulation without pre-built skills, demonstrations or privileged scene state, on which any Code-as-Policy harness can be compared or analyzed systematically. To build it, we adapt 25 RoboTwin 2.0 tasks with diverse manipulation demands, instructions aligned with hidden verifiers, feasibility confirmed by reference solutions through the same API, and checkpoints for trajectory-level analysis of intermediate progress.

 

W2 / Q1. Comparison with pre-built skills

We thank the reviewer for this insightful suggestion. We added a Code-as-Policy baseline with pre-built skills: CaP-X (Fu et al.) in M2, its multi-turn setting with image feedback, run with the same agent (GPT-6 Astra, Codex CLI), budgets and verifier on all 25 tasks in the three scenes of W5 (225 attempts). Its skills wrap perception and grasping: get_object_pose localizes objects with SAM3, sample_grasp_pose proposes grasps with Contact-GraspNet, and goto_pose moves by inverse kinematics.

Task successLocalization (SAM3)Valid pose proposalsSubgoal checkpoint coverageSpatial checkpoint coverage
CaP-X M237.8%91.1%84.8%52.5%75.1%
Ours72.0%NA89.4%78.2%88.1%

The skills find the right object and propose plannable poses, but not where to act on it: SAM3 localizes the named object in 91.1% of first queries and 84.8% of proposed poses are feasible, yet the arm reaches only 75.1% of spatial checkpoints (88.1% for ours) and completes 52.5% of subgoals (78.2%), nearly halving success (37.8% vs. 72.0%).

W3. Synthesis into broader takeaways and future directions

We thank the reviewer for this suggestion. We now synthesize the results into a single process analysis over all three scenes (W5; 2,025 attempts): where each attempt stops, and whether the agent recognizes that it has failed.

Using the checkpoints of Section 4.3, we record the stage at which each attempt ends:

  • Relevant regions not reached. A subgoal was never completed, and the gripper did not reach the region where each task object must be grasped or pressed.
  • Reached, subgoals incomplete. The gripper reached every such region, but a subgoal was never completed.
  • Subgoals completed, final state failed. Every subgoal was completed at some point, but the verifier's conditions did not hold when the attempt ended.
  • Success. The hidden verifier accepted the attempt.

A further column gives overclaims (a success claim on a failed attempt, Section 4.4) among failed attempts with an explicit claim.

ConfigurationRelevant regions not reachedReached, subgoals incompleteSubgoals completed, final state failedSuccessOverclaims
GPT-6 Astra (Codex CLI)4.0%21.3%2.7%72.0%39% (22/56)
Opus 5 (Claude Code)19.6%32.4%4.0%44.0%49% (36/74)
Opus 5 (Reference)16.0%35.6%6.2%42.2%49% (41/84)
GPT-5.6 Sol (Reference)45.3%32.9%1.8%20.0%63% (27/43)
Gemini 3.6 Flash (Reference)50.7%32.9%1.3%15.1%91% (62/68)
Kimi K3 (Reference)53.3%29.8%4.0%12.9%75% (18/24)
Qwen 3.8 Max (Reference)50.7%32.0%4.9%12.4%17% (10/59)
Grok 4.6 (Reference)64.0%26.2%0.4%9.3%25% (9/36)
Sonnet 5 (Reference)72.9%21.3%1.8%4.0%36% (4/11)
All (2,025 attempts)41.8%29.4%3.0%25.8%50% (229/455)

Weaker configurations fail earlier. Among attempts that do not complete every subgoal, 58–77% of those of the six weaker configurations never reach the relevant regions, against 16–38% for the top three. Completion judgment is a separate difficulty: half of the failed attempts with an explicit claim are overclaims (229/455), from 17% (Qwen 3.8 Max) to 91% (Gemini 3.6 Flash), although these two configurations have similar success rates.

Takeaways. Completion is hard to judge from vision alone: in scan_object, where the alignment tolerance (2.5 cm from the scanner's axis) is hard to observe, 34 of the 38 explicit claims on failed attempts are overclaims. A natural next step is to test whether agents understand their own execution, by aligning their accounts with the recorded robot actions. For agentic robotics, coding agents already serve as a capable brain; the open question is which hands can execute their plans more efficiently.

W4. Overlap among pick-and-place tasks

We thank the reviewer for this suggestion; broadening task diversity is valuable. We would like to clarify that the 11 tasks share grasp-and-release primitives but form a difficulty gradient and probe different capabilities. Over all nine configurations and three scenes, success ranges from 54% (place_mouse_pad) through 35% (stack_blocks_three) and 22% (place_bread_skillet) to 4% (hanging_mug).

The tasks also stress different abilities: precise alignment and size comparison (mouse pad, block ranking), ordered stacking (blocks, bowls), target orientation (shoes), bimanual coordination (bread into a held skillet, cans into a box), and regrasping with the other arm before a precise hang (mug).

 

W5 / Q2. More object configurations per task

We thank the reviewer for this suggestion, which makes the benchmark considerably more compelling. We added two new scenes (A and B) to each task and ran all nine configurations with three attempts per scene, for 2,025 attempts in total. Each new scene uses a new RoboTwin scene seed, which re-samples the object layout, and all 50 change the object appearance (via model swap or recolour) and the table and wall textures (from RoboTwin 2.0's unseen texture set).

ConfigurationOriginal sceneNew scene ANew scene BAll three
GPT-6 Astra (Codex CLI)54/7556/7552/75162/225 (72.0%)
Opus 5 (Claude Code)33/7536/7530/7599/225 (44.0%)
Opus 5 (Reference)37/7530/7528/7595/225 (42.2%)
GPT-5.6 Sol (Reference)12/7514/7519/7545/225 (20.0%)
Gemini 3.6 Flash (Reference)15/759/7510/7534/225 (15.1%)
Kimi K3 (Reference)9/7511/759/7529/225 (12.9%)
Qwen 3.8 Max (Reference)11/7510/757/7528/225 (12.4%)
Grok 4.6 (Reference)9/757/755/7521/225 (9.3%)
Sonnet 5 (Reference)2/753/754/759/225 (4.0%)
Total182/675176/675164/675522/2,025 (25.8%)

The ranking is stable across scenes: for any two scenes, at least 30 of the 36 configuration pairs keep the same order (Kendall's τ 0.70–0.89, exact p ≤ 0.0083). GPT-6 Astra stays at 69–75% in every scene, and the two Opus 5 configurations remain within 1.8 points overall (44.0% vs. 42.2%) while swapping order between scenes. Results thus show a degree of cross-scene generalization rather than reflecting a single fixed scene. We will merge the new scenes into the main results, updating relative sections.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.