acceptodds
Under review as a conference paper at ICLR 2027

EmbodiedSWE: Coding Agents for Long Horizon Dexterous Robotics

Abstract

We study coding agents for long-horizon, dexterous robotics and ask whether their solutions can provide scalable supervision for learning general robot policies. To test this, we develop EmbodiedSWE-Bench, an agent-native simulation benchmark spanning contact-rich manipulation, deformable objects, and long-horizon tasks requiring up to half an hour of continuous interaction. We find that frontier coding agents can solve complex long-horizon tasks and transfer prior solutions across both tasks and embodiments. We also design supporting tools that help agents more effectively solve these tasks. However, the resulting solutions require substantial iterative interaction and are typically specialized to individual task instances. We therefore introduce EmbodiedSWE-Gen, which expands a single solution from coding agent into large diverse trajectories for training a VLA. Finally, we present a pipeline that automatically generates diverse, verifiable simulation environments and explore improving coding agents through reinforcement learning. Together, our framework casts coding agents as solvers, teachers, and students in a scalable robot-learning loop.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robotics, Coding Agents

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ZmJiOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies coding agents as both robotic task solvers and sources of policy-training supervision. It introduces EmbodiedSWE-Bench, comprising 28 simulation tasks spanning contact-rich assembly, deformable objects, liquids, and locomotion with manipulation. Agents iteratively develop control programs with simulator access, and the paper evaluates frontier systems, solution transfer, and additional development tools. EmbodiedSWE-Gen then diversifies verified programs into demonstrations for VLA fine-tuning. The experiments show improved policy performance with more generated data and broader diversification, together with a preliminary real-robot lamp-disassembly demonstration using simulation-only fine-tuning data.

Critical strengths
  1. Substantial benchmark breadth and challenging physical interactions: The task collection extends beyond simple pick-and-place to threading, assembly, cutting, deformable manipulation, and long-horizon execution, with support for five embodiments (Section 2 and Appendix B). This is a useful setting for studying iterative robot-program development rather than only evaluating a fixed learned policy.

  2. Useful connection between coding agents and scalable robot supervision: Unlike human-seeded demonstration expansion such as MimicGen [1], the proposed pipeline starts from coding-agent-produced solutions. Its scene, strategy, phase, dynamics, and visual diversification, together with open-loop replay verification, turns successful programs into reusable training data rather than treating the code as the final deployed policy (Section 4.1 and Appendix D).

  3. Meaningful downstream evidence beyond agent leaderboards: The VLA experiments include three training seeds and equal-sized training sets for the diversification comparison. Agent-aided data improves the mean held-out rubric score from 0.066 to 0.233 in Table 1. The reported two complete real-robot successes in ten trials also provide preliminary evidence that the generated supervision is useful beyond simulation, although the result is still limited in scope.

References

[1] Ajay Mandlekar, Soroush Nasiriany, et al. MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations. Conference on Robot Learning (CoRL), PMLR 229:1820–1864, 2023.

Critical weaknesses
  1. Benchmark uncertainty and audit reliability are not sufficiently quantified: The main solver and tool experiments use one development run per task–model condition; the reported SEM across tasks does not measure variability across independent agent attempts on the same task. Appendix C.1 also uses Opus 5 to audit reward hacking and assigns zero to flagged runs, but does not report agreement with an independent human audit. Since these decisions change the reported scores substantially for some systems, a small audit-validation study and repeated development runs on a representative subset would strengthen the benchmark's reliability.

  2. The transfer and tool experiments contain important attribution confounds: Cross-embodiment transfer compares Kinova/xArm7 runs with a Franka solution hint against no-hint runs on Franka, rather than no-hint runs on the target embodiment (Section 3.2 and Table 20). The effect of the hint is therefore confounded with embodiment difficulty. Similarly, tool-enabled runs receive two GPUs versus one for the standard harness. Table 7 shows higher mean rubric scores, but Opus 5's full-task success drops from 50% to 43%, while GPT-5.6 Sol remains at 18%. These results support improved partial progress, not a demonstrated increase in full-task solving attributable to the tools alone.

  3. The policy-distillation results do not yet establish transfer of the benchmark's full long-horizon capability: Appendix E, Table 22 relaxes some success criteria: clearing one organic object or inserting one pen can count as success instead of completing the original multi-object task. The 69% VLA result should therefore not be interpreted as success on the unchanged benchmark objectives. The real-robot experiment is also a single task with 2/10 complete successes, and Appendix F states that the checkpoint was selected using real-robot performance. This is compatible with simulation-only demonstration fine-tuning, but it is not a transfer evaluation entirely independent of real-world feedback. These distinctions need clearer prominence in the main conclusions.

Questions for authors
  1. How well do the reward-hacking labels agree with independent human judgments on a balanced sample of accepted and rejected programs? Can a small representative task subset be repeated to estimate development-run variability, separately from variation across tasks?

  2. Can the authors add target-embodiment no-hint controls on a small subset of the transfer tasks? For the tool study, please distinguish improvements in rubric score from full-task success and clarify how much of the gain could come from the additional GPU budget.

  3. Can the existing VLA rollouts also be scored using the original full-task criteria? For the real-robot experiment, how many trials/checkpoints were used for selection, and were the ten reported evaluation trials separate from those selection trials?

6 comments

We thank the reviewer for the careful reading of our paper and the constructive comments. We appreciate the recognition of the benchmark's breadth and physical difficulty, of the connection between coding agents and scalable robot supervision, and of the downstream VLA evidence beyond agent leaderboards. We respond to each point below.

Benchmark uncertainty and audit reliability are not sufficiently quantified: The main solver and tool experiments use one development run per task–model condition; the reported SEM across tasks does not measure variability across independent agent attempts on the same task. [...] Since these decisions change the reported scores substantially for some systems, a small audit-validation study and repeated development runs on a representative subset would strengthen the benchmark's reliability.

Can a small representative task subset be repeated to estimate development-run variability, separately from variation across tasks?

Thank you for the suggestion. We will provide full multi-seed evaluation across all benchmark tasks and models in the final version of the paper. For this rebuttal, we present preliminary results from three independent runs on six tasks with Opus 5 and GPT-5.6 Sol, yielding 36 runs across assembly, packing, and puzzle tasks. Each run uses the same baseline harness, a four-hour budget, and one RTX 4090. The table reports final-submission rubric scores as mean ± standard deviation across the three independent attempts.

TaskDifficultyCategoryOpus 5GPT-5.6 Sol
Allen boltMediumAssembly0.680 ± 0.2810.558 ± 0.026
BulbMediumAssembly0.667 ± 0.0000.921 ± 0.062
PC motherboardMediumAssembly0.337 ± 0.1410.126 ± 0.002
PC RAMEasyAssembly0.278 ± 0.1920.333 ± 0.167
SpatulaEasyPuzzle0.600 ± 0.3460.267 ± 0.115
Tool packingMediumPacking0.733 ± 0.2310.733 ± 0.231

The average within-task/model standard deviation is 0.150.

Appendix C.1 also uses Opus 5 to audit reward hacking and assigns zero to flagged runs, but does not report agreement with an independent human audit. Since these decisions change the reported scores substantially for some systems, a small audit-validation study [...] would strengthen the benchmark's reliability.

How well do the reward-hacking labels agree with independent human judgments on a balanced sample of accepted and rejected programs?

Thank you for the suggestion. We manually re-verified each of the 29 runs flagged by the Opus 5 audit [and 29 randomly sampled accepted runs], and the human judgments agree with Opus 5's labels in all cases. We will report this validation in the appendix.

Reviewer_L6o2OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

EmbodiedSWE studies coding agents as robot-program developers and demonstration generators. EmbodiedSWE-Bench provides 28 simulated robotics tasks with source and state access, configurable controllers, and isolated grading. Language models operating through Claude Code or Codex iteratively write, execute, and revise Python control programs, delivered through a solve(env) entry point. EmbodiedSWE-Gen expands successful programs through scene, strategy, phase, dynamics, and visual variation to produce demonstrations for a separate VLA finetuning stage. The paper evaluates six model/harness configurations, studies solution transfer and development tools, and demonstrates downstream policy learning in simulation and on a real robot.

Critical strengths
  1. A substantial platform for developing programs for difficult physical tasks. The 28 tasks cover precision assembly, cutting, deformable objects, liquids, and locomotion with manipulation across five embodiments. The environments, controllers, graders, and program-development interface give researchers concrete tasks on which to evaluate robot coding agents. Across six model/harness configurations, solved-task counts range from 3 to 23 under a four-hour budget, providing useful task-level evidence about current capabilities and failures.

  2. An instructive connection between program development and robot learning, supported by downstream results. The workflow turns expensive code development into reusable supervision: successful programs generate varied demonstrations for policy training. With 1,000 demonstrations per task and condition, agent-aided diversification improves all six tasks on common held-out configurations, raising the mean rubric score from 0.066 to 0.233. Increasing demonstrations from 10 to 400 per task raises SmolVLA's mean in-distribution success rate from about 18% to 69%. This makes the approach a useful direction for collecting training data for contact-intensive manipulation.

Critical weaknesses
  1. The breadth of the paper leaves important methodological and evaluation details underdeveloped. The paper covers a benchmark, coding-agent evaluation, development tools, program transfer, demonstration generation, and VLA learning, leaving substantial gaps in the main-text account of the core workflow. In particular, the roles of Claude Code/Codex and the underlying model, the code-development loop, the delivered control program, and its connection to subsequent policy training need a clearer explanation. A compact end-to-end example would make the approach much easier to understand and reproduce.

  2. The data engine also needs a stronger comparative and cost evaluation. The script-only comparator uses domain randomization, noise, and small initial-state perturbations, while GenSim2 already supports semantic task and scene variation and solver generation. A direct comparison would better establish the benefit of editable programs and intermediate starting states. Figures 4 and 8 and Tables 8, 12, and 13 provide useful token and spending information, but the reader still lacks a consolidated cost per accepted demonstration, including unsuccessful attempts, simulation, replay, rendering, and human effort. Table 13 reports API spending for solved runs.

Questions for authors
  1. Network and search tools: Section 2.2 restricts network access to the model API and Python package index. Did the evaluation also disable the harnesses' built-in search tools, including provider-hosted search accessed through the allowed model API? Please specify the harness versions and effective tool configurations used in the reported experiments, and how this restriction was verified.

  2. End-to-end cost: Beyond the token and partial spending results already reported, what were the total API cost and GPU time for producing the training datasets, including failed generation attempts, replay verification, and rendering? Please report the cost per accepted physical trajectory and the human effort required for task construction, repair, and auditing.

  3. Quantifying data diversity: Could you report, for each task, the numbers of distinct scenes, strategies, and starting phases and their sample distributions? Please distinguish independent physical trajectories from visual rerenderings of the same trajectory, and provide a direct measure of diversity, such as behavioral coverage or trajectory near-duplicate rates, alongside dataset size and downstream performance.

6 comments

We thank the reviewer for the thorough review and the detailed, insightful feedback. We appreciate the recognition of EmbodiedSWE-Bench as a substantial platform for developing programs for difficult physical tasks, and of the connection between program development and robot learning supported by our downstream results. Our point-by-point responses follow.

The breadth of the paper leaves important methodological and evaluation details underdeveloped. The paper covers a benchmark, coding-agent evaluation, development tools, program transfer, demonstration generation, and VLA learning, leaving substantial gaps in the main-text account of the core workflow. In particular, the roles of Claude Code/Codex and the underlying model, the code-development loop, the delivered control program, and its connection to subsequent policy training need a clearer explanation. A compact end-to-end example would make the approach much easier to understand and reproduce.

Thank you for the suggestion. The overall workflow is shown in Figure 1 (bottom), the evaluation protocol is in Section 2.2, and the data engine is detailed in Appendix D. The supplementary code also walks through the full pipeline. We agree that the main text should connect these pieces more clearly. In brief:

Task → Develop → Grade → Diversify → Generate Dataset → Train VLA

  1. Task: the coding agent (a model such as Fable 5.1 running in Claude Code) receives a task instruction and the simulator.
  2. Develop: it writes a solve(env) program, runs it in simulation, inspects the results, and revises the program.
  3. Grade: each submitted program is scored offline on new scene instances.
  4. Diversify: from a verified program, the agent edits the scene and solution code to create new scenes, strategies, and starting states.
  5. Generate Dataset: the programs are rolled out in parallel simulation, and only successful, replay-verified trajectories are kept. These are then re-rendered under varied lighting, textures, and camera poses.
  6. Train VLA: these trajectories are used to fine-tune a VLA, which is the policy we evaluate and deploy.

We will provide a concrete end-to-end example that walks through this workflow on one task in the appendix.

The data engine also needs a stronger comparative and cost evaluation. The script-only comparator uses domain randomization, noise, and small initial-state perturbations, while GenSim2 already supports semantic task and scene variation and solver generation. A direct comparison would better establish the benefit of editable programs and intermediate starting states.

Thank you for pointing us to these related works; we will discuss them more explicitly in the main text.

How our pipeline relates to existing ones. Most existing data-generation pipelines augment some combination of scene, objects, initial conditions, physical parameters, and rendering. Some focus on one dimension. For example, the MimicGen family [1–3] and DemoGen [4] adapt a few human demonstrations to new object poses. Others vary several at once: GenSim and GenSim2 [5, 6], RoboGen [7], and RoboCasa [8] generate new tasks, scenes, and objects. Because their inputs do not contain the full task structure, these pipelines often need extra stages to recover it, such as annotating subtask segments in human demonstrations, segmenting objects to retarget a demonstration, or labeling per-asset keypoints for a motion planner (GenSim2).

Our pipeline is designed for a different setting, in which a simulated scene and a working code-as-policy solution already exist, as in EmbodiedSWE-Bench. Here, the coding agent wrote the solution itself and has full access to the simulator source and state. The task structure is therefore already explicit in its code: phase boundaries, grasp targets, step order, and success checks. Diversification becomes a direct edit to the scene or solution code rather than an inference problem. This has three benefits:

  1. No human labor, so it scales. Every stage, from solving the task to diversification, generation, and verification, is carried out by the coding agent. In principle, the pipeline needs no human demonstrations, keypoint labels, or reward design, so it scales with compute and API budget rather than with human effort.
  2. More kinds of variation. Besides scene, initial-state, dynamics, and visual variation, the agent can change how the task is solved: reordering steps, switching grasps, adding recovery behaviors, or changing the overall plan. Figure 3 shows examples of each, with the original solution next to the diversified versions.
  3. Easy to use and low cost. For the slice task, diversification took one 1.6-hour agent session ($45), and the full pipeline costs about $0.02 in API fees and 0.6 GPU-minutes per accepted trajectory (see our response to point 3).

On a direct comparison with GenSim2. GenSim2's solvers rely on human-labeled keypoints and a hand-built motion planner, which do not exist for our tasks and cannot express their contact-rich, long-horizon behavior, such as threading, cutting, and deformable manipulation. As with RL (our response to Reviewer 7tu6, point 3), we expect this to yield no successful demonstrations on our tasks, so there would be no training data to compare.

Complementary to existing pipelines. We see existing pipelines as complementary rather than competing. Our agent works with richer oracle information than existing pipelines: the simulator source, the full simulator state, and its own solution code. Any variation used by an existing pipeline can therefore be added to ours as a new setting, without the extra stages those pipelines need to recover this information. Within our setting, Table 1 compares against the most relevant alternative: script-only augmentation of the same verified program, with equal data size.

[1] MimicGen (Mandlekar et al., 2023); [2] DexMimicGen (Jiang et al., 2025); [3] SkillMimicGen (Garrett et al., 2024); [4] DemoGen (Xue et al., 2025); [5] GenSim (Wang et al., 2024); [6] GenSim2 (Hua et al., 2024); [7] RoboGen (Wang et al., 2024); [8] RoboCasa (Nasiriany et al., 2024).

Reviewer_7tu6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors present a thorough look into how coding agents are able to perform on long-horizon dexterous tasks and how they can be leveraged to create diverse training trajectories for bootstrapping generalist policies. The authors create EmbodiedSWE-Bench, a dataset of daily-life tasks that could take tens of minutes to solve. Some tasks include contact-intensive manipulation, working with deformable objects or liquids, and cutting. Frontier coding agents are then evaluated on this benchmark and the more recent LLMs achieve high success rates. The solutions generated by the coding agents are then used for VLA finetuning, resulting in improved performance with increasing amounts of demonstrations as well as improved out-of-distribution performance compared to domain randomization.

Critical strengths
  • The EmbodiedSWE-Bench is a well-thought out and useful benchmark for long-horizon and contact-rich manipulation robotic tasks, including assembly, packing, puzzle, deformable objects, liquid manipulation, cutting, and locomotion with manipulation.
  • This work serves as an example of how coding agents can be leveraged as a reliable source of training demonstrations, which could scale up robot policy learning. I particularly appreciated the experiments that showed that these generated demonstrations improve VLA policy performance and generalization to held-out tasks. I also greatly appreciated the frontier agent failure mode analysis.
Critical weaknesses
  • The success rates for the generalization experiments as well as half the real-world experiments are around 0.2-0.3 which leaves much to be desired. Especially since only one baseline was used as comparison for each set.
  • The authors perform 3 seed evaluations for the transfer-learning experiments but the benchmark success rate only uses one run per model-task pair. A single run here may not be sufficient to estimate model performance due to significant variability in agent outputs.
Questions for authors
  • Could the authors run the primary coding agent evaluation with multiple seeds?
  • For the generalization experiments, the agent-aided experiments score only 0.23 on average, leaving large gaps for improvement. Do the authors have any ideas on how to better leverage coding agents to drive up the generalization of held-out variations?
  • Could the authors compare agent-aided demonstrations with RL-based demonstrations in terms of downstream policy performance and computational cost?
4 comments

We thank the reviewer for the time and effort devoted to evaluating our work and for the constructive feedback. We appreciate the recognition of EmbodiedSWE-Bench as a well-designed benchmark for long-horizon, contact-rich manipulation, as well as of our VLA experiments with agent-generated demonstrations and our analysis of frontier-agent failure modes. We address each comment below.

The success rates for the generalization experiments as well as half the real-world experiments are around 0.2-0.3 which leaves much to be desired. Especially since only one baseline was used as comparison for each set.

For the generalization experiments, the agent-aided experiments score only 0.23 on average, leaving large gaps for improvement. Do the authors have any ideas on how to better leverage coding agents to drive up the generalization of held-out variations?

Thank you for raising this; we agree there is room to improve. We see two main reasons for the current numbers, both tied to our limited compute:

  1. Small policy model. We distill into SmolVLA (~450M parameters), trained for at most 100k steps on one GPU. More trainable capacity already helps: also fine-tuning the language model (not just the action expert) lifts bulb success from 41% to 75%.
  2. Limited scene diversity. Given our generation budget, each generalization set covers a modest number of scene variations, so the held-out settings remain fairly far from what the policy saw in training.

Neither is a limit of the method itself. With more GPU compute we can distill into larger VLAs and train longer; with more coding-agent budget and GPU compute for simulation, we can generate demonstrations in many more scenes and focus on the steps where the policy fails. Another approach we tried after the submission: once the policy is evaluated or deployed, we ask the coding agent to look at the failed cases and generate more demonstrations for them, much like a researcher would when improving a VLA. This has helped in our early experiments, but we do not yet have a rigorous evaluation for it.

The authors perform 3 seed evaluations for the transfer-learning experiments but the benchmark success rate only uses one run per model-task pair. A single run here may not be sufficient to estimate model performance due to significant variability in agent outputs.

Could the authors run the primary coding agent evaluation with multiple seeds?

Thank you for the suggestion. We will provide full multi-seed evaluation across all benchmark tasks and models in the final version of the paper. For this rebuttal, we present preliminary results from three independent runs on six tasks with Opus 5 and GPT-5.6 Sol, yielding 36 runs across assembly, packing, and puzzle tasks. Each run uses the same baseline harness, a four-hour budget, and one RTX 4090. The table reports final-submission rubric scores as mean ± standard deviation across the three independent attempts.

TaskDifficultyCategoryOpus 5GPT-5.6 Sol
Allen boltMediumAssembly0.680 ± 0.2810.558 ± 0.026
BulbMediumAssembly0.667 ± 0.0000.921 ± 0.062
PC motherboardMediumAssembly0.337 ± 0.1410.126 ± 0.002
PC RAMEasyAssembly0.278 ± 0.1920.333 ± 0.167
SpatulaEasyPuzzle0.600 ± 0.3460.267 ± 0.115
Tool packingMediumPacking0.733 ± 0.2310.733 ± 0.231

The average within-task/model standard deviation is 0.150.

Could the authors compare agent-aided demonstrations with RL-based demonstrations in terms of downstream policy performance and computational cost?

Thank you for the question. The main reason we did not use RL-based demonstrations is that no RL setting produced a single successful episode on our multi-stage manipulation tasks. Even with about a week of expert tuning, the best episodes only reach about one third of full progress. As a result, there are no successful RL demonstrations to train a downstream policy on. We will make this comparison clearer in the main text.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.