acceptodds
Desk rejected at ICLR 2027

Grounded-Dreamer: Robot Learning from World Model Synthetic Data with Physical Grounding

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Scaling robot manipulation policies requires large amounts of diverse, action-labeled data, yet collecting such data in the real world remains expensive and difficult to scale. Existing methods either depend on human teleoperation, rely on simulation with limited visual realism, or use human videos that lack embodiment-specific annotations. In this paper, we introduce Grounded-Dreamer, a framework for synthesizing physically grounded robot manipulation data with video world models while avoiding human teleoperation demonstrations. Grounded-Dreamer combines the visual realism of generative video models with simulator physics engines. Starting from a small set of real images depicting robot task initial states, our method automatically reconstructs task-relevant objects and camera geometry, aligns them in simulation, and collects physically valid trajectories. These trajectories are used to post-train both a video world model and an inverse dynamics model, enabling the generation of visually realistic robot manipulation videos and corresponding robot actions from real images. To filter physically implausible rollouts, Grounded-Dreamer replays generated trajectories in the reconstructed simulation and retains only those that successfully complete the tasks. The real-world robot policies are trained solely on the grounded synthetic trajectories without any human demonstrations. Empirical results in real-world tasks demonstrate that Grounded-Dreamer can synthesize physically grounded data and substantially facilitate policy learning without requiring human teleoperation effort.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot Manipulation, World Models, Synthetic Data Generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Z7z1OpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2(+1)
Presentation
2(+1)
Soundness
2(+1)
Read review
Paper summary

This paper presents Grounded-Dreamer, a robot-data synthesis framework intended to eliminate the need for human teleoperation demonstrations. The central idea is to combine the visual realism of a pretrained video world model with the physical constraints of a reconstructed simulator. Starting from a small number of real images, the method reconstructs task-relevant objects and camera geometry, automatically collects trajectories in simulation, fine-tunes a video world model and inverse dynamics model (IDM), generates synthetic robot videos/actions, and finally replays the inferred actions in simulation to reject task-failing trajectories. Policies are then trained solely or jointly on the resulting synthetic trajectories.

Critical strengths

The authors generate varied backgrounds, illumination, and textures and report substantially better OOD results than a small fixed real dataset. This supports one genuine advantage of generative synthetic data: environmental diversity can be obtained much more cheaply than by rebuilding real environments.

The paper is really weak that i did not see any strengths.

Critical weaknesses
  1. “Physical grounding” is weaker than the paper's terminology suggests This is my primary concern. The final verification procedure appears to verify whether the IDM-derived action trajectory succeeds when replayed in the reconstructed simulator.
    However, the training sample consists of a generated visual trajectory paired with those inferred actions. Simulation replay does not necessarily establish that the generated images themselves are physically consistent with the replayed simulator trajectory. For example, suppose the video generator hallucinates an object displacement, while the IDM happens to produce an action sequence that successfully performs the task in simulation. The sample passes verification, despite potentially having inconsistent image/action/state transitions. Thus:
SimSuccess(a1:T)⇏o1:TWM≈o1:Tphysics.\text{SimSuccess}(a_{1:T}) \not\Rightarrow o^{WM}_{1:T}\approx o^{physics}_{1:T}.

This distinction is particularly important because the downstream policy learns the mapping from generated observations to actions. The paper would be significantly stronger if it measured, for accepted rollouts, agreement between world-model-generated states and simulation-replayed states, e.g. object 6D pose, end-effector pose, contact events, or segmentation trajectories. At present I would describe the method as simulation-verified action trajectories, rather than “strict physical grounding.” 2. The claimed autonomy hides substantial task-specific structure The manuscript repeatedly emphasizes data generation “without human demonstrations” and “purely autonomous data generation.” However, the simulator trajectory generator receives:

  • reference code templates defining trajectory-generation logic,
  • modular execution stages,
  • IK-based end-effector control,
  • high-level task descriptions,
  • object metadata.
    For articulated objects, the pipeline additionally uses predefined connectivity rules and joint specifications.
    This is very different from autonomous acquisition of manipulation skills from a few images. Human demonstrations are eliminated, but human task engineering may simply have moved from trajectory collection into task templates, connectivity definitions, reward/success criteria, object scaling, and task setup. The paper should quantify the human effort needed to introduce a completely new task. In particular:
  • Is a separate reference-code template written for each task?
  • Which parts of the code shown to the LLM are task independent?
  • Who defines success conditions?
  • Who provides articulated-object connectivity?
  • How is real object scale obtained?
  • How much manual correction is needed after SAM3D reconstruction? Without this accounting, “no human demonstrations” is accurate, but “autonomous data generation” feels overstated.
  1. The experimental comparison does not isolate whether the world model itself is necessary The framework already has a simulator capable of producing 100–150 successful action trajectories per task. It additionally has powerful image-generation machinery for realistic appearance. Therefore, an important missing question is: Why is video-world-model generation preferable to simply retaining the simulator actions and translating/randomizing the simulator observations into realistic images?

The current “Sim-only” baseline uses raw simulator observations and is therefore a weak baseline for this question. I would like to see comparisons against at least:

  • simulator trajectories + conventional domain randomization;
  • simulator trajectories + Nano Banana image translation;
  • simulator trajectories + interaction masking;
  • simulator actions paired with photorealistically transformed rendered frames;
  • same Cosmos/IDM pipeline without replay filtering;
  • a DreamGen-style world-model trajectory generation baseline using the same backbone. These would determine whether performance comes from the world-model dynamics, image augmentation, simulation trajectory generation, filtering, or some combination. This is especially important because the closest conceptual prior work, DreamGen, is discussed in Related Work but is not experimentally compared.
  1. The verification ablation is not sufficient to establish the central claim Table 4 varies the proportion of verified versus sub-optimal data and finds that more verified data gives higher success.
    This is directionally useful, but it does not answer the strongest questions about verification:
  • What fraction of generated trajectories passes the verifier?
  • What are false-positive and false-negative rates?
  • Does passing simulation replay predict real-world success?
  • What happens with 0% verification / all raw world-model generations?
  • Are total dataset sizes held constant in Table 4?
  • How is “Sub-optimal” sampled?
  • Does the verifier merely select trajectories that are easier? The strongest experiment would randomly sample generated trajectories, label each as pass/fail in simulation, then execute a subset on hardware and report
P(real success∣sim pass)P(\text{real success}\mid \text{sim pass})

versus

P(real success∣sim fail).P(\text{real success}\mid \text{sim fail}).

That would directly validate the paper's core hypothesis. 5. Statistical evidence is weak Most reported evaluations use only 20 trials per condition. Thus each 5% change corresponds to a single rollout. Differences such as 55% versus 65%, or 60% versus 70%, cannot be interpreted strongly without confidence intervals. Moreover, the paper does not appear to report multiple independent policy-training seeds. This becomes particularly problematic for statements such as “matches or exceeds” the real-data baseline. The differences often lie well within the binomial uncertainty of 20 robot trials. At minimum, I recommend:

  • confidence intervals for success rates;
  • several independent training runs;
  • aggregate success with statistical tests or bootstrap intervals.
  1. The OOD evaluation is underspecified and may favor the proposed augmentation strategy The training pipeline explicitly varies backgrounds, tablecloths, and lighting using Nano Banana 2. The OOD experiment then evaluates environmental variation and reports very large improvements.
    It is unclear how far the test conditions lie outside the augmentation distribution. A more convincing OOD test would separately evaluate:
  • unseen object instances;
  • novel object geometry;
  • unseen object poses;
  • camera viewpoint changes;
  • illumination;
  • backgrounds;
  • distractors;
  • dynamics changes;
  • novel task compositions. A standard real-data baseline with conventional image augmentation should also be included. There is additionally a numerical inconsistency: Table 2 gives the Real-only DP results as 10,5,5,0,5,5,010,5,5,0,5,5,0, whose average is approximately 4.3%, whereas the text states 7.1%.
  1. “Scaling laws” is an overstatement Section 4.4 calls the experiment a study of “scaling laws,” but only two tasks and roughly five dataset sizes are considered. Performance simply increases and then saturates around 150 trajectories.
    This is useful as a data-scaling study, but it is not sufficient evidence for a scaling law. No functional relationship is fitted and no extrapolation across models/tasks is provided. I recommend renaming this section to something like Effect of Synthetic Dataset Size unless considerably larger-scale evidence is added.
  2. Computational efficiency and scalability need considerably more analysis The framework uses eight A800 GPUs. Cosmos fine-tuning requires approximately 74 GB per GPU, 1,400–2,000 iterations, and the IDM is also trained using eight GPUs.
    The paper also reports that initial-frame synthesis succeeds only about 65% of the time.
    Given that data scalability is a central motivation, the manuscript should report:
  • wall-clock training time per task;
  • generation time per trajectory;
  • image-generation API/model cost;
  • verification rejection rate;
  • number of attempted generations required to obtain 50/150/250 accepted samples;
  • whether Cosmos and IDM are separately fine-tuned for every task;
  • amortized cost when adding another task. The claim that synthetic trajectories can be generated at “near-zero cost” is hard to reconcile with the stated compute requirements unless “cost” refers specifically to human labor.
Questions for authors

Questions for the Authors

  1. What exactly does simulation verification verify? Do you quantitatively compare generated visual states to replayed simulator states, or only check task success after replaying IDM actions?
  2. How much task-specific human engineering is required? Please report the effort needed to add a completely new task, including reference-code templates, success predicates, object connectivity, scaling, and simulator corrections.
  3. Is Cosmos fine-tuned separately for each task? Is the IDM task specific? If so, how does compute scale with the number of tasks?
  4. What percentage of generated trajectories passes replay verification for each task? The appendix appears to report roughly 45% success during generation, but a per-task breakdown would be useful.
  5. Can the authors report real-world success conditioned on verifier pass/fail? This would be the cleanest evidence that simulation verification provides useful physical grounding.
  6. Why is there no stronger sim-to-real baseline? In particular, I would like to see the same automatically generated simulator actions combined with photorealistic image translation/domain randomization, without using a video world model.
  7. Can the authors include a fully unverified world-model-data condition? Table 4 stops at a mixed dataset and therefore does not directly show the benefit of filtering all raw generated data.
  8. How are OOD conditions constructed, and are they guaranteed not to overlap the image-augmentation distribution?
  9. Can the authors provide confidence intervals and multiple policy-training seeds?
  10. Why does the OOD DP average reported in the text appear inconsistent with Table 2? The table values average to about 4.3%, not 7.1%.
2 comments
AuthorsOpenReview ↗

Response to Reviewer Z7z1

Thank you for your questions.

  1. Scope of physical verification. We verify whether the actions inferred from a generated video complete the task when replayed in the reconstructed simulator. Replay starts from the object poses used to construct the initial frame (Sec. 3.4). Fig. 4 illustrates the distinction: the generated video depicts a successful grasp, but the gripper misses the object in replay. Measuring frame-level consistency would require estimating object poses and contact events throughout the generated video and aligning them with simulator states. This would add substantial implementation effort, computation, and time. Such measurements would address a separate question about the physical accuracy of the video. They are unnecessary for the task-level action check our pipeline is designed to perform.

  2. Definition and sampling of sub-optimal trajectories. A trajectory is labeled sub-optimal when its inferred actions fail the task during simulation replay. Failure can arise from an error in the world-model generation, an error in IDM action extraction, or both. To construct the mixtures in Table 4, we randomly sample sub-optimal trajectories from the pool of replay failures.

  3. Effect of replay filtering. As the fraction of replay-passing training trajectories increased from 50% to 80% to 100%, mean real-robot policy success across the seven tasks in Table 4 rose from 21.4% to 32.9% to 42.9% for DP. For π₀.₅, the corresponding rates were 24.3%, 42.1%, and 55.0%.

  4. Relation to unfiltered world-model data. We chose the 50% condition because the raw world-model outputs contained roughly equal proportions of replay successes and failures. An unfiltered dataset retains both groups. A dataset with 0% replay-passing trajectories would contain only failures. The 50% mixture in Table 4 therefore approximates the composition of the raw dataset, although we used replay labels to construct the mixture.

  5. Human input and template reuse. Generating GD training trajectories requires neither per-trajectory human teleoperation nor a seed set of teleoperated demonstrations. Initial setup typically uses one scene overview and one to four object views (Appendix A). We also measure relevant object properties, including physical dimensions. The task templates are prepared once and reused in subsequent data collection.

  6. Automated trajectory collection. The reference templates provide the execution stages and inverse-kinematics control logic. The task description specifies the objective and object semantics. Once the scene is configured, the agent reads object poses and physical properties directly from Isaac Sim, then generates and executes the collection scripts. This procedure produced 100–150 simulation trajectories per task (Sec. 3.2).

  7. Object reconstruction. For rigid objects, we rescale the SAM3D meshes to match measured real-world dimensions. For articulated objects, SAM3D separates the components. We manually annotate their connectivity and measure and specify the joint parameters. The agent then uses these specifications to assemble the components in simulation (Sec. 3.1).

  8. Generation cost. Once a task is configured, each additional trajectory can be generated without further human teleoperation. This is the labor saving intended by “near-zero cost.” The pipeline still incurs computational costs. The 0.5 seconds per trajectory reported in Appendix E.3 measures IDM action extraction alone and excludes image synthesis, video generation, and simulation replay.

  9. OOD evaluation. We manually selected additional real-world test scenes and checked that none appeared in either the real-robot training data or the GD training data. All policies were evaluated on the same unseen scenes. The holdout is at the scene level: individual factors such as lighting and tablecloth texture can vary along dimensions also covered by the training augmentations.

  10. Correction to Sec. 4.3. The seven Real-only DP entries in Table 2 average to 4.3%. We will correct the value in Sec. 4.3 from 7.1% to 4.3%.

After reading the author's response, i think this paper is really weak because most of the method purposed by this paper can be easily finished by agentic real2sim like astra and fable right now. And the novelity becomes very limited. I can understand the author's feeling, the emerging spatial reasoning ability of large model destroy and changing the acadmic community of robotic significantly, but it does not means that the paper with limited contribution should be accepted.

But I do think all score of 1 being too strict. so i will change my score accordingly.

Reviewer_bw4zOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Grounded-Dreamer reconstructs a simulation environment from real-world reference images and automatically collects simulated trajectories to fine-tune a video world model and an inverse dynamics model. It then generates task videos from synthesized initial frames, recovers action trajectories, and verifies task success through simulation replay. Experiments on seven manipulation tasks show that policies trained on the generated datasets achieve success rates comparable to those trained on real-world human demonstrations.

Critical strengths
  1. The proposed Interaction-Aware Masking strategy helps the video world model focus on realistic robot-object interactions while preserving task-irrelevant backgrounds
  2. The proposed closed-loop simulation verification replays IDM-recovered trajectories to filter out actions that fail to complete the task in simulation.
Critical weaknesses
  1. The method primarily combines existing models into a data generation workflow. A closely related idea of verifying video generated actions through simulation replay has been explored in RoboCurate: Harnessing Diversity with Action-Verified Neural Trajectory for Robot Learning. The authors should discuss this work and clarify the methodological differences in related work.
  2. The proposed closed-loop simulation verification appears to assess task success of the recovered actions. However, successful replay in simulation does not necessarily rule out incorrect occlusions or contacts in the generated videos. Such inconsistencies may reduce the usefulness of the retained demonstrations for policy training.
  3. The OOD evaluation protocol is insufficiently specified. Section 4.3 describes training-time visual augmentation but does not clearly define the test conditions or their differences from both the real and augmented GD training distributions.
  4. The evaluated tasks are relatively limited in complexity and horizon. It remains unclear whether the LLM agent based automated data collection pipeline can reliably generate trajectories for more complex, long-horizon tasks.
Questions for authors
  1. Co-training uses 55 trajectories, whereas Real-only and GD-only use 50. Could the authors add a matched-size comparison to isolate the benefit of mixing real and synthetic data?
  2. Beyond action feasibility, is it necessary to compare robot-object interactions in generated and simulated videos to filter out inconsistent occlusions and contacts?
  3. Could the authors specify the OOD test conditions and clarify whether they are unseen in both the real and augmented GD training data?
Reviewer_mikpOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents Grounded-Dreamer, a pipeline for synthesizing robot training data. The system reconstructs task-relevant simulation environments from a small set of real reference images, automatically collects simulated trajectories, and uses them to fine-tune a video world model and an inverse dynamics model (IDM). It then randomizes object poses in simulation and constructs initial frames using real backgrounds and an image-editing model. The video model generates manipulation videos, and the IDM extracts action sequences. These trajectories are replayed in the reconstructed simulator and filtered according to task success criteria before being used to train robot policies. Real-robot experiments show that policies trained only on the filtered synthetic data approach the performance of policies trained on real demonstrations on the evaluated tasks, although they still perform worse on most tasks.

Critical strengths
  1. The pipeline connects realistic appearance generation with action verification in simulation, offering a valuable route toward reducing human teleoperation demonstrations for target tasks. Its value lies in converting generated videos into action-labeled data checked for task success, for downstream robot policy training.
  2. The method is presented clearly. Experiments use both DP and π₀.₅ policy backbones, and ablations of interaction-aware masking and replay filtering demonstrate the effects of key components on downstream performance.
Critical weaknesses
  1. End-to-end efficiency and the cost of adapting to new tasks are not sufficiently quantified. Although the appendix provides model training configurations, it does not fully report the human setup time required for a new task, total computation time, or the acceptance rates and failure-handling procedures at each stage. In particular, the amount of human input required for object rescaling, joint specifications, and data-collection templates remains unclear, as does whether the world model and IDM need to be trained separately for each task. The appendix also reports an initial-frame synthesis success rate of approximately 65%, but does not sufficiently explain how failed samples are identified and handled. These details affect the assessment of the pipeline's degree of automation and scalability.

  2. Evidence for long-horizon task capabilities remains limited. The paper describes its tasks as spanning short- to long-horizon manipulation. However, in the demonstration videos, the drawer in Put in Drawer is already open, and the tools in Sweep Trash and Slice Sausage are already held by the gripper. These settings primarily evaluate local operations from prepared initial states, rather than complete tasks involving multiple stages such as tool acquisition, articulated-object manipulation, and subsequent operations. This does not diminish the control challenges of sweeping and slicing themselves, but the current demonstrations provide limited evidence for long-horizon, multi-stage capabilities. The authors should clarify whether these initial conditions apply to all quantitative evaluations, describe the stages of each task, and qualify the corresponding claims.

Questions for authors
  1. For a new task, which steps require human configuration, correction, or filtering? Are the world model and IDM trained separately for each task? Could the authors report the total time from reference images to a specified number of accepted trajectories, along with acceptance rates and failure-handling procedures for initial-frame generation, video generation, and simulation replay filtering?
  2. Do the prepared initial conditions in the demonstration videos, namely an already-open drawer and already-grasped tools, apply to all quantitative evaluations? How is a long-horizon task defined, and which stages must be completed sequentially in each task?
  3. In the OOD comparisons, are the real demonstrations augmented with background, tablecloth, and lighting variations comparable to those used for the synthetic data? If not, how can the contribution of the data-generation pipeline itself be distinguished from the benefit of increased visual diversity in the training data?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.