- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies coding agents as both robotic task solvers and sources of policy-training supervision. It introduces EmbodiedSWE-Bench, comprising 28 simulation tasks spanning contact-rich assembly, deformable objects, liquids, and locomotion with manipulation. Agents iteratively develop control programs with simulator access, and the paper evaluates frontier systems, solution transfer, and additional development tools. EmbodiedSWE-Gen then diversifies verified programs into demonstrations for VLA fine-tuning. The experiments show improved policy performance with more generated data and broader diversification, together with a preliminary real-robot lamp-disassembly demonstration using simulation-only fine-tuning data.
Critical strengths
-
Substantial benchmark breadth and challenging physical interactions: The task collection extends beyond simple pick-and-place to threading, assembly, cutting, deformable manipulation, and long-horizon execution, with support for five embodiments (Section 2 and Appendix B). This is a useful setting for studying iterative robot-program development rather than only evaluating a fixed learned policy.
-
Useful connection between coding agents and scalable robot supervision: Unlike human-seeded demonstration expansion such as MimicGen [1], the proposed pipeline starts from coding-agent-produced solutions. Its scene, strategy, phase, dynamics, and visual diversification, together with open-loop replay verification, turns successful programs into reusable training data rather than treating the code as the final deployed policy (Section 4.1 and Appendix D).
-
Meaningful downstream evidence beyond agent leaderboards: The VLA experiments include three training seeds and equal-sized training sets for the diversification comparison. Agent-aided data improves the mean held-out rubric score from 0.066 to 0.233 in Table 1. The reported two complete real-robot successes in ten trials also provide preliminary evidence that the generated supervision is useful beyond simulation, although the result is still limited in scope.
References
[1] Ajay Mandlekar, Soroush Nasiriany, et al. MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations. Conference on Robot Learning (CoRL), PMLR 229:1820–1864, 2023.
Critical weaknesses
-
Benchmark uncertainty and audit reliability are not sufficiently quantified: The main solver and tool experiments use one development run per task–model condition; the reported SEM across tasks does not measure variability across independent agent attempts on the same task. Appendix C.1 also uses Opus 5 to audit reward hacking and assigns zero to flagged runs, but does not report agreement with an independent human audit. Since these decisions change the reported scores substantially for some systems, a small audit-validation study and repeated development runs on a representative subset would strengthen the benchmark's reliability.
-
The transfer and tool experiments contain important attribution confounds: Cross-embodiment transfer compares Kinova/xArm7 runs with a Franka solution hint against no-hint runs on Franka, rather than no-hint runs on the target embodiment (Section 3.2 and Table 20). The effect of the hint is therefore confounded with embodiment difficulty. Similarly, tool-enabled runs receive two GPUs versus one for the standard harness. Table 7 shows higher mean rubric scores, but Opus 5's full-task success drops from 50% to 43%, while GPT-5.6 Sol remains at 18%. These results support improved partial progress, not a demonstrated increase in full-task solving attributable to the tools alone.
-
The policy-distillation results do not yet establish transfer of the benchmark's full long-horizon capability: Appendix E, Table 22 relaxes some success criteria: clearing one organic object or inserting one pen can count as success instead of completing the original multi-object task. The 69% VLA result should therefore not be interpreted as success on the unchanged benchmark objectives. The real-robot experiment is also a single task with 2/10 complete successes, and Appendix F states that the checkpoint was selected using real-robot performance. This is compatible with simulation-only demonstration fine-tuning, but it is not a transfer evaluation entirely independent of real-world feedback. These distinctions need clearer prominence in the main conclusions.
Questions for authors
-
How well do the reward-hacking labels agree with independent human judgments on a balanced sample of accepted and rejected programs? Can a small representative task subset be repeated to estimate development-run variability, separately from variation across tasks?
-
Can the authors add target-embodiment no-hint controls on a small subset of the transfer tasks? For the tool study, please distinguish improvements in rubric score from full-task success and clarify how much of the gain could come from the additional GPU budget.
-
Can the existing VLA rollouts also be scored using the original full-task criteria? For the real-robot experiment, how many trials/checkpoints were used for selection, and were the ten reported evaluation trials separate from those selection trials?
6 comments
Benchmark uncertainty and audit reliability are not sufficiently quantified: The main solver and tool experiments use one development run per task–model condition; the reported SEM across tasks does not measure variability across independent agent attempts on the same task. [...] Since these decisions change the reported scores substantially for some systems, a small audit-validation study and repeated development runs on a representative subset would strengthen the benchmark's reliability.
Can a small representative task subset be repeated to estimate development-run variability, separately from variation across tasks?
Thank you for the suggestion. We will provide full multi-seed evaluation across all benchmark tasks and models in the final version of the paper. For this rebuttal, we present preliminary results from three independent runs on six tasks with Opus 5 and GPT-5.6 Sol, yielding 36 runs across assembly, packing, and puzzle tasks. Each run uses the same baseline harness, a four-hour budget, and one RTX 4090. The table reports final-submission rubric scores as mean ± standard deviation across the three independent attempts.
| Task | Difficulty | Category | Opus 5 | GPT-5.6 Sol |
|---|---|---|---|---|
| Allen bolt | Medium | Assembly | 0.680 ± 0.281 | 0.558 ± 0.026 |
| Bulb | Medium | Assembly | 0.667 ± 0.000 | 0.921 ± 0.062 |
| PC motherboard | Medium | Assembly | 0.337 ± 0.141 | 0.126 ± 0.002 |
| PC RAM | Easy | Assembly | 0.278 ± 0.192 | 0.333 ± 0.167 |
| Spatula | Easy | Puzzle | 0.600 ± 0.346 | 0.267 ± 0.115 |
| Tool packing | Medium | Packing | 0.733 ± 0.231 | 0.733 ± 0.231 |
The average within-task/model standard deviation is 0.150.
Appendix C.1 also uses Opus 5 to audit reward hacking and assigns zero to flagged runs, but does not report agreement with an independent human audit. Since these decisions change the reported scores substantially for some systems, a small audit-validation study [...] would strengthen the benchmark's reliability.
How well do the reward-hacking labels agree with independent human judgments on a balanced sample of accepted and rejected programs?
Thank you for the suggestion. We manually re-verified each of the 29 runs flagged by the Opus 5 audit [and 29 randomly sampled accepted runs], and the human judgments agree with Opus 5's labels in all cases. We will report this validation in the appendix.
We thank the reviewer for the careful reading of our paper and the constructive comments. We appreciate the recognition of the benchmark's breadth and physical difficulty, of the connection between coding agents and scalable robot supervision, and of the downstream VLA evidence beyond agent leaderboards. We respond to each point below.