acceptodds
Withdrawn from ICLR 2027

EmbodiedSmith: Scaling Embodied Data through Recursive Self-Improvement Flywheel in Simulation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

28 views

Abstract

Scaling robotic foundation models requires diverse training data and reliable evaluation environments. Simulation offers a scalable solution, yet existing generation pipelines remain constrained by predefined assets and skills, a disconnect between scene generation and task generation, and limited support for complex embodiments and physics. We introduce EmbodiedSmith, a framework for scalable embodied data generation through recursive self-improvement (RSI). EmbodiedSmith unifies asset, scene, and task generation in a pipeline that supports autonomous creation and language-driven customization. Its core is an agentic refinement loop: scene generation anticipates downstream task requirements, while task generation guides targeted scene edits, allowing scenes and tasks to iteratively improve one another. This joint refinement improves task generation success, including for long-horizon tasks. The framework further supports mobile manipulators, humanoids, and dexterous hands, as well as interactions involving deformable objects and fluids, broadening the range of behaviors and physical phenomena represented in generated data. Together, these capabilities provide a flexible simulation engine for both robot pretraining and evaluation. Extensive experiments validate the quality, diversity, and generation efficiency of the resulting data, while downstream policy experiments demonstrate that increased data diversity improves generalization.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Generative AI, Robotic Data

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_EFWYOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents an embodied synthetic data generation framework, named EmbodiedSmith. EmbodiedSmith turns the task and robot description into a scene graph, retrieves or generates the objects, and converts the graph into object poses. It then turns existing robot skills into an execution plan. After execution, the framework will inspect failures and revise the execution plan or change the scene instead. Experiments evaluate scene quality, demonstration generation on 16 fixed-base and mobile manipulation tasks, and downstream policy training on generated demonstrations. The results show improved demonstration-generation success better than previous methods.

Critical strengths
  • The system tries to address a practical challenge that exists in the simulation generation: a generated scene ought to be valid for a robot to execute, and a proposed task in the scene should be able to find solutions. Combining task grounding, motion planning, execution feedback, and explicit acceptance checks provides a useful approach to automated data collection. The held-out-seed evaluation and inclusion of failed generation sessions in ASR are positive aspects of the protocol.
Critical weaknesses
  • The major issue of the paper is that the experiments do not sufficiently isolate the claimed scene–task self-improvement contribution: Appendix C.4 states that scene assets and object layouts remain fixed during the main demonstration-generation evaluation. The ablations therefore establish the importance of execution-feedback-based plan repair and robot placement validation, but do not directly establish the benefit of the proposed mutual scene–task refinement loop. Furthermore, the scene baselines retain different asset pipelines, and the policy experiment compares randomized with fixed-layout data rather than with another randomized collection method. Spatial domain randomization can help the robot policy generalize to new layouts has been studied before. Therefore, these results support system utility but leave the source of the improvement insufficiently resolved.

  • The technical specification and validation are insufficient for several central claims as well. Section 3.1 names Beam-DFS layout optimization without adequately specifying its objective, candidate generation, search procedure, or connection to visual scene-graph refinement. The described task generator composes predefined atomic skills, leaving the claimed creation of new skills unclear. Likewise, the presented embodiment demonstrations and physics descriptions do not establish human-level dexterity or improved physical fidelity. These gaps make it difficult to assess the methodological contribution over existing task-conditioned, simulation-ready generation systems such as SceneSmith and EmbodiedGen V2.

  • The paper writing and figure generation seem to use widely used AI and are hard to follow. For example, the figure 1 and 2 have various arrows and hard to follow the full pipeline. Moreover, it is hard to tell which parts are new compared with previous methods, since the pipeline lists many components and most of them have been discussed in previous methods.

Questions for authors
  • How does the generated scene image inform scene-graph refinement? Please specify the layout solver’s objective, pose-candidate generation, search and pruning procedure, and treatment of reachability constraints.

  • Which quantitative experiment isolates execution-driven scene editing? A comparison with scene editing enabled versus disabled, using identical assets, initial scenes, tasks, and generation budgets, would materially affect my assessment of the central contribution.

  • Does the system generate new atomic skill implementations, or compose and parameterize the primitives in Table 5? What quantitative evidence supports the claims concerning human-level dexterity and deformable/fluid interaction fidelity?

  • Can the downstream gains be compared against conventional layout randomization using the same demonstration budget and execution backend? This would help establish the benefit specific to the proposed generation framework.

Reviewer_3j8nOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a framework that generates simulation scenes and robot demonstration from a language description and a robot specification.

A VLM plans the scene structure and a solver places assets. An agent then composes predefined atomic skills into an executable task plan which is run in simulation and repaired from execution feedback.

Experiments evaluate scene quality, demonstration-generation success on 16 tasks, and policy training with π0.5 and FastWAM. They report higher demonstration success than with SceneSmith and EmbodiedGen V2 scenes.

Critical strengths
  1. The paper targets an important timely problem of generating executable simulation data for robot foundation models.
  2. The task-generation harness is well designed where the LLM only proposes plans, while a deterministic verifier checks, executes, and gates repairs under fixed success criteria.
  3. Demonstration generation results are strong, 67% vs 21% average ASR for SceneSmith and EmbodiedGen V2 scenes (Table 2)
Critical weaknesses
  1. Novelty of the task generation pipeline is unclear The core of the method (sec 3.2 an LLM agent composing predefined skills into a plan, executing in simulation, and revising from execution and VLM feedback) is essentially the pipeline of RoboTwin 2.0 (Chen et al. 2025). It also uses ASR / Top-5 ASR and the same one-shot vs. feedback ablation. Task conditioned scene plus task generation with LLMs is in RoboGen (Wang et al., 2024) and GenSim2 (Hua et al. 2024). None of these are cited, and sec 2 covers only scene generation and humanoid work. The new part appears to be integration with room-scale scene generation but the paper should state this explicitly and compare against RoboTwin 2.0’s planning loop in Tables 2–3.

  2. The diversity claim is not validated Data diversity is the first listed contribution (sec 1), but nothing in the experiments measures it. There is no diversity metric over the generated scenes. The paper also lists deformables and fluids as part of this diversity, yet all tasks in Tables 2, 4 and 8 involve rigid or articulated objects (with Sponge Wiping in Table 4 as the only possible exception). There are no deformable- or fluid-specific task validations.

given the limited novelty over prior work and the gap between what is claimed in Sec. 1 and what the experiments actually test, I don’t think the paper meets the quality bar in its current form

Questions for authors

Which VLM versions were used to run the baselines in Tables 1 and 2? It is unclear whether the gains come from the method or from a newer model.

Reviewer_G5fAOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

EmbodiedSmith presents an agent-based framework for generating simulation environments and robot demonstrations from task specifications and robot configurations. It integrates asset retrieval and generation, scene layout construction, geometric and reachability checks, skill-based task planning, simulation execution, and feedback-driven refinement. Programmatic acceptance checks are used to validate task outcomes and recorded trajectories, while embodiment-specific execution adapters support different robot configurations. The evaluation covers scene quality, demonstration generation across 16 fixed-base and mobile manipulation tasks, workflow ablations, and downstream policy training using clean and randomized datasets. The reported results suggest that the pipeline improves demonstration generation reliability and that randomized training data improve policy performance under layout variation.

Critical strengths
  1. The paper addresses a meaningful practical bottleneck and incorporates useful programmatic validation.

The distinction between a visually plausible scene and an environment in which a robot can actually complete a task is important for simulation-based data generation. EmbodiedSmith addresses this issue by connecting scene construction to robot reachability, executable skill plans, and simulation feedback. The verification structure is particularly valuable: acceptance depends on task predicates, required intermediate events, and trajectory integrity, while task-success criteria remain fixed during refinement. These mechanisms make the proposed system more credible as a source of usable demonstrations.

  1. The evaluation connects environment quality to downstream learning, and the architecture supports practical reuse.

The paper evaluates scene quality, demonstration generation, and the usefulness of the resulting data for policy learning. The equal-duration clean-versus-mixed comparison is a useful design choice because it provides evidence for the benefit of randomized training data without simply increasing the training-data duration. In addition, separating the shared skill DAG from embodiment-specific execution adapters is a reasonable architectural choice for adapting the system to different robots. Together, these aspects give the work practical value as an integrated data-generation framework.

Critical weaknesses
  1. The refinement process is insufficiently characterized to explain what is being improved and why the method succeeds.

The improvements in demonstration generation are plausible, and Table 3 provides useful evidence that execution feedback and placement validation matter. Section 4.2 and Figure 5 also discuss failure causes such as inappropriate support heights, object scale, clutter, and limited reachability. However, the paper does not provide a systematic account of how frequently these failures occur or how the proposed interventions resolve them.

In particular, it remains unclear what fraction of initial proposals succeed immediately, which failure categories dominate, how many revisions are typically needed, what is changed during those revisions, and how often an initially failed candidate is successfully recovered. The maximum budget of 30 rounds and cumulative token counts do not reveal the actual distribution or magnitude of repairs. Similarly, the collection success rates in Table 9 do not distinguish initially successful cases from cases recovered through targeted repair or obtained after resampling.

This gap is consequential because Appendix C.4 states that the main fixed-scene evaluation keeps scene assets and object layouts unchanged, permitting revisions to the task plan and initial robot placement. These experiments directly support feedback-based plan refinement, but do not isolate the additional contribution of mutual scene–task refinement, which is central to the paper’s framing. The current description and aggregate results therefore provide limited insight into the nature and scope of the claimed recursive self-improvement.

  1. The resulting data diversity and the broader scaling and capability claims are insufficiently substantiated.

Appendix D describes a useful hierarchy of asset, layout, and observation randomization. However, specifying the dimensions that can be randomized does not establish the diversity actually achieved in the accepted dataset. The paper does not sufficiently quantify the realized coverage of assets, layouts, task or skill compositions, and interaction types, or how this coverage changes after validation and refinement.

This is particularly important because the collection procedure progressively replaces episode conditions, layouts, and eventually asset compositions when failures persist. Such resampling is a reasonable production strategy, but without rejection and replacement statistics it is difficult to determine how much of the improvement comes from repairing challenging configurations and how much comes from selecting configurations that are easier to execute. The effect of this process on the final data distribution remains unclear.

The downstream comparison uses two scene types and ten tasks with a fixed 20-hour data budget. It supports the usefulness of randomized data within those settings, but provides limited evidence about how coverage, generation yield, cost, or policy performance evolve as the production scale increases. Likewise, the humanoid, dexterous-hand, deformable-object, and fluid capabilities are described or illustrated, but their reliability, diversity, and downstream learning utility are not quantitatively established. These gaps limit the support for the paper’s broader claims concerning scaling and the breadth of the generated data.

Questions for authors
  1. Failure and repair accounting — clarification and supporting analysis.

Can the authors provide a breakdown of first-attempt success, initial failure categories, revision-count distributions, and the types and magnitudes of edits actually applied? In particular, what fraction of initially failed candidates are recovered through targeted repair, and what fraction are discarded or replaced through resampling? A correspondence between failure types and successful corrective actions would help explain why refinement works.

  1. Scene–task co-refinement — clarification of the evaluated mechanism.

Appendix C.4 states that scene assets and object layouts remain fixed during the main quantitative evaluation. Which experiments directly evaluate the contribution of scene editing to task generation? Is there a comparison that separates plan refinement, robot-placement refinement, and joint scene–plan refinement from matched initial configurations and under comparable computational budgets?

  1. Realized diversity and scaling — evidence relevant to the central claims.

What diversity is present in the final accepted dataset in terms of unique assets, layouts, task or skill compositions, and interaction types? How does this distribution compare with the initial and rejected candidates? Please also clarify what evidence supports scaling beyond the fixed-duration downstream comparison, such as changes in accepted-data yield, coverage, or generation cost as the generation budget increases.

  1. Additional embodiments and physics — clarification of demonstrated scope.

What quantitative evidence is available for the humanoid, dexterous-hand, deformable-object, and fluid capabilities? Which of these capabilities contribute to the evaluated training datasets, and which are currently supported primarily by implementation descriptions or qualitative demonstrations?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.