- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2(+1)
- Presentation
- 2(+1)
- Soundness
- 2(+1)
Read review
Paper summary
This paper presents Grounded-Dreamer, a robot-data synthesis framework intended to eliminate the need for human teleoperation demonstrations. The central idea is to combine the visual realism of a pretrained video world model with the physical constraints of a reconstructed simulator. Starting from a small number of real images, the method reconstructs task-relevant objects and camera geometry, automatically collects trajectories in simulation, fine-tunes a video world model and inverse dynamics model (IDM), generates synthetic robot videos/actions, and finally replays the inferred actions in simulation to reject task-failing trajectories. Policies are then trained solely or jointly on the resulting synthetic trajectories.
Critical strengths
The authors generate varied backgrounds, illumination, and textures and report substantially better OOD results than a small fixed real dataset. This supports one genuine advantage of generative synthetic data: environmental diversity can be obtained much more cheaply than by rebuilding real environments.
The paper is really weak that i did not see any strengths.
Critical weaknesses
- “Physical grounding” is weaker than the paper's terminology suggests
This is my primary concern.
The final verification procedure appears to verify whether the IDM-derived action trajectory succeeds when replayed in the reconstructed simulator.
However, the training sample consists of a generated visual trajectory paired with those inferred actions. Simulation replay does not necessarily establish that the generated images themselves are physically consistent with the replayed simulator trajectory. For example, suppose the video generator hallucinates an object displacement, while the IDM happens to produce an action sequence that successfully performs the task in simulation. The sample passes verification, despite potentially having inconsistent image/action/state transitions. Thus:
This distinction is particularly important because the downstream policy learns the mapping from generated observations to actions. The paper would be significantly stronger if it measured, for accepted rollouts, agreement between world-model-generated states and simulation-replayed states, e.g. object 6D pose, end-effector pose, contact events, or segmentation trajectories. At present I would describe the method as simulation-verified action trajectories, rather than “strict physical grounding.” 2. The claimed autonomy hides substantial task-specific structure The manuscript repeatedly emphasizes data generation “without human demonstrations” and “purely autonomous data generation.” However, the simulator trajectory generator receives:
- reference code templates defining trajectory-generation logic,
- modular execution stages,
- IK-based end-effector control,
- high-level task descriptions,
- object metadata.
For articulated objects, the pipeline additionally uses predefined connectivity rules and joint specifications.
This is very different from autonomous acquisition of manipulation skills from a few images. Human demonstrations are eliminated, but human task engineering may simply have moved from trajectory collection into task templates, connectivity definitions, reward/success criteria, object scaling, and task setup. The paper should quantify the human effort needed to introduce a completely new task. In particular: - Is a separate reference-code template written for each task?
- Which parts of the code shown to the LLM are task independent?
- Who defines success conditions?
- Who provides articulated-object connectivity?
- How is real object scale obtained?
- How much manual correction is needed after SAM3D reconstruction? Without this accounting, “no human demonstrations” is accurate, but “autonomous data generation” feels overstated.
- The experimental comparison does not isolate whether the world model itself is necessary The framework already has a simulator capable of producing 100–150 successful action trajectories per task. It additionally has powerful image-generation machinery for realistic appearance. Therefore, an important missing question is: Why is video-world-model generation preferable to simply retaining the simulator actions and translating/randomizing the simulator observations into realistic images?
The current “Sim-only” baseline uses raw simulator observations and is therefore a weak baseline for this question. I would like to see comparisons against at least:
- simulator trajectories + conventional domain randomization;
- simulator trajectories + Nano Banana image translation;
- simulator trajectories + interaction masking;
- simulator actions paired with photorealistically transformed rendered frames;
- same Cosmos/IDM pipeline without replay filtering;
- a DreamGen-style world-model trajectory generation baseline using the same backbone. These would determine whether performance comes from the world-model dynamics, image augmentation, simulation trajectory generation, filtering, or some combination. This is especially important because the closest conceptual prior work, DreamGen, is discussed in Related Work but is not experimentally compared.
- The verification ablation is not sufficient to establish the central claim
Table 4 varies the proportion of verified versus sub-optimal data and finds that more verified data gives higher success.
This is directionally useful, but it does not answer the strongest questions about verification:
- What fraction of generated trajectories passes the verifier?
- What are false-positive and false-negative rates?
- Does passing simulation replay predict real-world success?
- What happens with 0% verification / all raw world-model generations?
- Are total dataset sizes held constant in Table 4?
- How is “Sub-optimal” sampled?
- Does the verifier merely select trajectories that are easier? The strongest experiment would randomly sample generated trajectories, label each as pass/fail in simulation, then execute a subset on hardware and report
versus
That would directly validate the paper's core hypothesis. 5. Statistical evidence is weak Most reported evaluations use only 20 trials per condition. Thus each 5% change corresponds to a single rollout. Differences such as 55% versus 65%, or 60% versus 70%, cannot be interpreted strongly without confidence intervals. Moreover, the paper does not appear to report multiple independent policy-training seeds. This becomes particularly problematic for statements such as “matches or exceeds” the real-data baseline. The differences often lie well within the binomial uncertainty of 20 robot trials. At minimum, I recommend:
- confidence intervals for success rates;
- several independent training runs;
- aggregate success with statistical tests or bootstrap intervals.
- The OOD evaluation is underspecified and may favor the proposed augmentation strategy
The training pipeline explicitly varies backgrounds, tablecloths, and lighting using Nano Banana 2. The OOD experiment then evaluates environmental variation and reports very large improvements.
It is unclear how far the test conditions lie outside the augmentation distribution. A more convincing OOD test would separately evaluate:
- unseen object instances;
- novel object geometry;
- unseen object poses;
- camera viewpoint changes;
- illumination;
- backgrounds;
- distractors;
- dynamics changes;
- novel task compositions. A standard real-data baseline with conventional image augmentation should also be included. There is additionally a numerical inconsistency: Table 2 gives the Real-only DP results as , whose average is approximately 4.3%, whereas the text states 7.1%.
- “Scaling laws” is an overstatement
Section 4.4 calls the experiment a study of “scaling laws,” but only two tasks and roughly five dataset sizes are considered. Performance simply increases and then saturates around 150 trajectories.
This is useful as a data-scaling study, but it is not sufficient evidence for a scaling law. No functional relationship is fitted and no extrapolation across models/tasks is provided. I recommend renaming this section to something like Effect of Synthetic Dataset Size unless considerably larger-scale evidence is added. - Computational efficiency and scalability need considerably more analysis
The framework uses eight A800 GPUs. Cosmos fine-tuning requires approximately 74 GB per GPU, 1,400–2,000 iterations, and the IDM is also trained using eight GPUs.
The paper also reports that initial-frame synthesis succeeds only about 65% of the time.
Given that data scalability is a central motivation, the manuscript should report:
- wall-clock training time per task;
- generation time per trajectory;
- image-generation API/model cost;
- verification rejection rate;
- number of attempted generations required to obtain 50/150/250 accepted samples;
- whether Cosmos and IDM are separately fine-tuned for every task;
- amortized cost when adding another task. The claim that synthetic trajectories can be generated at “near-zero cost” is hard to reconcile with the stated compute requirements unless “cost” refers specifically to human labor.
Questions for authors
Questions for the Authors
- What exactly does simulation verification verify? Do you quantitatively compare generated visual states to replayed simulator states, or only check task success after replaying IDM actions?
- How much task-specific human engineering is required? Please report the effort needed to add a completely new task, including reference-code templates, success predicates, object connectivity, scaling, and simulator corrections.
- Is Cosmos fine-tuned separately for each task? Is the IDM task specific? If so, how does compute scale with the number of tasks?
- What percentage of generated trajectories passes replay verification for each task? The appendix appears to report roughly 45% success during generation, but a per-task breakdown would be useful.
- Can the authors report real-world success conditioned on verifier pass/fail? This would be the cleanest evidence that simulation verification provides useful physical grounding.
- Why is there no stronger sim-to-real baseline? In particular, I would like to see the same automatically generated simulator actions combined with photorealistic image translation/domain randomization, without using a video world model.
- Can the authors include a fully unverified world-model-data condition? Table 4 stops at a mixed dataset and therefore does not directly show the benefit of filtering all raw generated data.
- How are OOD conditions constructed, and are they guaranteed not to overlap the image-augmentation distribution?
- Can the authors provide confidence intervals and multiple policy-training seeds?
- Why does the OOD DP average reported in the text appear inconsistent with Table 2? The table values average to about 4.3%, not 7.1%.
2 comments
After reading the author's response, i think this paper is really weak because most of the method purposed by this paper can be easily finished by agentic real2sim like astra and fable right now. And the novelity becomes very limited. I can understand the author's feeling, the emerging spatial reasoning ability of large model destroy and changing the acadmic community of robotic significantly, but it does not means that the paper with limited contribution should be accepted.
But I do think all score of 1 being too strict. so i will change my score accordingly.
Response to Reviewer Z7z1
Thank you for your questions.
Scope of physical verification. We verify whether the actions inferred from a generated video complete the task when replayed in the reconstructed simulator. Replay starts from the object poses used to construct the initial frame (Sec. 3.4). Fig. 4 illustrates the distinction: the generated video depicts a successful grasp, but the gripper misses the object in replay. Measuring frame-level consistency would require estimating object poses and contact events throughout the generated video and aligning them with simulator states. This would add substantial implementation effort, computation, and time. Such measurements would address a separate question about the physical accuracy of the video. They are unnecessary for the task-level action check our pipeline is designed to perform.
Definition and sampling of sub-optimal trajectories. A trajectory is labeled sub-optimal when its inferred actions fail the task during simulation replay. Failure can arise from an error in the world-model generation, an error in IDM action extraction, or both. To construct the mixtures in Table 4, we randomly sample sub-optimal trajectories from the pool of replay failures.
Effect of replay filtering. As the fraction of replay-passing training trajectories increased from 50% to 80% to 100%, mean real-robot policy success across the seven tasks in Table 4 rose from 21.4% to 32.9% to 42.9% for DP. For π₀.₅, the corresponding rates were 24.3%, 42.1%, and 55.0%.
Relation to unfiltered world-model data. We chose the 50% condition because the raw world-model outputs contained roughly equal proportions of replay successes and failures. An unfiltered dataset retains both groups. A dataset with 0% replay-passing trajectories would contain only failures. The 50% mixture in Table 4 therefore approximates the composition of the raw dataset, although we used replay labels to construct the mixture.
Human input and template reuse. Generating GD training trajectories requires neither per-trajectory human teleoperation nor a seed set of teleoperated demonstrations. Initial setup typically uses one scene overview and one to four object views (Appendix A). We also measure relevant object properties, including physical dimensions. The task templates are prepared once and reused in subsequent data collection.
Automated trajectory collection. The reference templates provide the execution stages and inverse-kinematics control logic. The task description specifies the objective and object semantics. Once the scene is configured, the agent reads object poses and physical properties directly from Isaac Sim, then generates and executes the collection scripts. This procedure produced 100–150 simulation trajectories per task (Sec. 3.2).
Object reconstruction. For rigid objects, we rescale the SAM3D meshes to match measured real-world dimensions. For articulated objects, SAM3D separates the components. We manually annotate their connectivity and measure and specify the joint parameters. The agent then uses these specifications to assemble the components in simulation (Sec. 3.1).
Generation cost. Once a task is configured, each additional trajectory can be generated without further human teleoperation. This is the labor saving intended by “near-zero cost.” The pipeline still incurs computational costs. The 0.5 seconds per trajectory reported in Appendix E.3 measures IDM action extraction alone and excludes image synthesis, video generation, and simulation replay.
OOD evaluation. We manually selected additional real-world test scenes and checked that none appeared in either the real-robot training data or the GD training data. All policies were evaluated on the same unseen scenes. The holdout is at the scene level: individual factors such as lighting and tablecloth texture can vary along dimensions also covered by the training augmentations.
Correction to Sec. 4.3. The seven Real-only DP entries in Table 2 average to 4.3%. We will correct the value in Sec. 4.3 from 7.1% to 4.3%.