acceptodds
Under review as a conference paper at ICLR 2027

Learning Primitive Embodied World Models: Towards Scalable Robot Learning

Abstract

While video-generation-based embodied world models have gained increasing attention, their reliance on large-scale embodied interaction data remains a key bottleneck. The scarcity, difficulty of collection, and high dimensionality of em- bodied data fundamentally limit the alignment granularity between language and actions and exacerbate the challenge of long-horizon video generation—hindering generative models from achieving a "GPT moment" in the embodied domain. There is a naive observation: the diversity of embodied data far exceeds the relatively small space of possible primitive motions. Based on this insight, we propose a novel paradigm for world modeling–Primitive Embodied World Models (PEWM). By restricting video generation to fixed short horizons, our approach 1) enables fine- grained alignment between linguistic concepts and visual representations of robotic actions, 2) reduces learning complexity, 3) improves data efficiency in embodied data collection, and 4) decreases inference latency. By equipping with a modular Vision-Language Model (VLM) planner and a Start-Goal heatmap Guidance mech- anism (SGG), PEWM further enables flexible closed-loop control and supports compositional generalization of primitive-level policies over extended, complex tasks. Our framework leverages the spatiotemporal vision priors in video mod- els and the semantic awareness of VLMs to bridge the gap between fine-grained physical interaction and high-level reasoning, paving the way toward scalable, interpretable, and general-purpose embodied intelligence.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Models, Video Diffusion Models, Zero-Shot Policy Learning, Embodied Intelligence

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_GCYcOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Primitive Embodied World Models (PEWM), a framework that organizes robot demonstrations into short, semantically atomic motion primitives. A video diffusion model predicts primitive-level visual trajectories conditioned on language instructions and start–goal heatmaps. Executable end-effector trajectories are then recovered using a pretrained pose estimator and geometric transformations. A VLM planner composes these primitives for longer tasks. The framework is evaluated through RLBench experiments, real-robot manipulation, video-generation metrics, and component ablations.

Critical strengths
  • A coherent and reasonably comprehensive system. The paper connects primitive-level data collection and annotation, video generation, spatial grounding, pose extraction, and hierarchical execution into a concrete robotic pipeline.

  • A useful emphasis on data organization. Aligning short video segments with individual manipulation intents is a plausible way to improve language–motion alignment and facilitate reuse across tasks. Recovering trajectories without training a dedicated action decoder is also an interesting design choice.

  • Evaluation includes physical execution. Beyond generated-video examples, the paper reports real-robot task success, compositional evaluations, and ablations. These provide evidence that the framework has practical utility.

Critical weaknesses
  • The conceptual advance and resulting insight are limited. Most components build on established techniques, and the contribution appears primarily to be their integration around primitive-level data. The experiments do not clearly isolate whether the gains arise from semantic segmentation, additional annotations, spatial guidance, the VLM planner, or the overall training recipe.

  • The baseline comparisons are insufficiently controlled. Several simulation results are taken directly from prior papers without establishing equivalent training data and evaluation protocols. The supplementary comparison against Colosseum-derived results is particularly difficult to interpret. Recent WAM approaches, such as Fast-WAM and DreamZero, also deserve discussion and, where feasible, experimental comparison, given the paper’s claims about control, generalization, and efficiency.

  • The empirical evidence supports narrower conclusions than the paper claims. The main simulation evaluation covers nine RLBench tasks, with a modest average improvement over 4DWM and regressions on three tasks. Real-robot comparisons cover three tasks with 20 trials each. These results demonstrate feasibility, but provide limited support for broad claims about scalability, systematic compositional generalization, or data efficiency.

  • The efficiency reporting is internally inconsistent. The main text reports distilled generation at 12 FPS, whereas the appendix reports 32 frames in 16 seconds and approximately 31 seconds per primitive including pose estimation. The appendix also describes acceleration techniques as future work. These configurations need to be reconciled before the real-time deployment claim can be assessed.

Questions for authors
  • What is the independent benefit of semantic primitives? Can the authors compare semantic segmentation against fixed-length segmentation using the same demonstrations, backbone, annotation budget, and planner?

  • How are training and evaluation separated? Please clarify which tasks, objects, and action–object combinations are excluded from training, separately for the video model and VLM planner. What target-task supervision is used when the video model remains frozen?

  • Are the baseline protocols comparable? Please specify differences in training data, task variations, observations, execution budgets, and success criteria. Which exact Colosseum conditions produced the numbers used in Table 8?

  • Why is explicit video generation followed by pose extraction preferable to learned action prediction? A representative WAM baseline or a controlled action-decoder comparison would help establish the benefits and costs of this design.

  • Which configuration produces the reported real-robot results? Please reconcile the latency measurements and identify whether the distilled model was used in the success-rate evaluations. Report end-to-end latency, including planning, video generation, and pose extraction.

Reviewer_8P3UOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Primitive Embodied World Models (PEWM), a primitive-level video-generation framework for robotic learning. The method decomposes long-horizon robotic tasks into short, language-grounded action primitives. A hierarchical framework combining a Vision-Language Model (VLM) planner with Start-Goal Heatmap Guidance (SGG) enables compositional task execution, while 6-DoF trajectories extracted from generated videos are used for robot control. The authors also introduce Sim-Real Hybrid training and video generation acceleration techniques. Experiments on RLBench and real-world robotic manipulation tasks demonstrate promising task success, compositional generalization, and video generation efficiency.

Critical strengths
  1. The paper introduces a primitive-level world modeling framework that decomposes long-horizon robotic tasks into reusable, short-horizon semantic actions, providing a promising approach to compositional generalization.
  2. The method achieves substantial video generation efficiency, producing 32 frames in approximately 16 seconds (2.0 FPS), representing over a 40$\times$ speedup compared with 4DWM.
  3. By combining VLM planning, Start-Goal Heatmap Guidance, and Sim-Real Hybrid training, the approach achieves competitive performance in both simulated and real-world robotic tasks.
Critical weaknesses
  1. The static rollout visualizations in Figure 12 are not sufficiently informative. For example, in the ``Move the yellow toy to the top of the wooden cube'' task, the generated frames with Sim-Real Hybrid show little visible progression, making it difficult to assess the execution process and final outcome. Moreover, the visual differences between the results with and without Sim-Real Hybrid are not sufficiently apparent, making it difficult to clearly demonstrate the benefits of the proposed hybrid training strategy.
  2. Long-horizon task execution is a central contribution of PEWM, but the current evidence is largely qualitative, particularly in Figure 13. More comprehensive quantitative evaluations of long-horizon success rates, performance across different task lengths, and failure cases are needed.
  3. Although PEWM achieves strong results on several video quality metrics, Table 7 shows that its LPIPS (0.1647) and VIF (0.3188) are worse than those of Hunyuan I2V (0.1466 and 0.3529, respectively). The reasons for these differences and the potential trade-off between perceptual quality and physical consistency are not sufficiently analyzed.
Questions for authors
  1. Can the authors provide complete video rollouts corresponding to Figure 12 in the supplementary material to better demonstrate the execution process and the benefits of Sim-Real Hybrid training?
  2. Given the importance of long-horizon execution, can the authors report success rates across tasks of varying complexity and primitive sequence lengths, together with more detailed visualizations?
  3. Can the authors explain why PEWM underperforms some baselines on LPIPS and VIF in Table 7, despite achieving better results on several other metrics, and discuss the underlying quality trade-offs?
Reviewer_xEXFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

PEWM proposes organizing embodied video learning around short, language-grounded primitives. The system combines a DynamiCrafter-based video model trained with simulation and real data, a VLM primitive planner, start–goal heatmap guidance, and Gen6D-based end-effector trajectory extraction. Primitives are composed for longer tasks. Validation includes RLBench results, three real-robot tasks, component ablations, predicate–object composition tests, and video-quality/physical-consistency metrics. The paper also claims real-time distilled generation and gives a theorem intended to justify the expressivity of primitive compositions.

Critical strengths
  1. Primitive-level data organization and explicit spatial guidance form a plausible approach to reusable embodied video models. Combining high-level decomposition with short visual rollouts is a useful system direction, especially when robot demonstrations are limited.

  2. The evaluation includes actual execution rather than only generated-video appearance. Table 2 reports real-robot successes of 16/20, 14/20, and 13/20; Table 4 separately ablates guidance, the primitive planner, and simulation data (both on p. 8). Pose-extraction/post-processing tests and failures are reported in Appendix B.2 (pp. 18–19), while EPiCS criteria and inter-annotator agreement are given on p. 31. These provide useful diagnostic evidence, although the scale remains limited.

Critical weaknesses
  1. The central theoretical justification does not follow from its assumptions. Theorem A.4 claims density in the space of arbitrary continuous trajectories, but A.1–A.3 do not guarantee that a primitive can connect arbitrary nearby endpoints within an arbitrarily short duration. The proof invokes A.2 for precisely that unsupported existence step (pp. 16–17). It then infers uniform tracking from continuity and short intervals without a uniform bound on the selected primitive trajectories; endpoint proximity does not exclude an intermediate excursion. Approximate endpoints also do not automatically give a continuous concatenation. The cardinality argument further counts templates while overlooking continuously parameterized instances, and does not establish the stated information-theoretic optimality. This is a substantive correctness issue, not merely missing proof detail. The theorem should be withdrawn or reformulated on a suitable reachable trajectory class with sufficient tracking and concatenation assumptions.

  2. The principal empirical claims need a consistent mapping to the evaluated model and data protocol. Section 3.3 reports four-step causal distillation and 12 FPS (p. 6), whereas Table 9 reports 32 frames in 16 seconds, or 2 FPS (p. 25), and Table 10 gives approximately 31 seconds per primitive, including about 12 seconds for pose estimation (p. 26). Appendix H.5 explicitly describes unaccelerated inference and leaves distillation to future work (pp. 25–26). These could be different configurations; I cannot infer that the reported distilled system does not exist. The unresolved issue is which configuration produces the task-success results. There is also a real-observation feedback loop described in Appendix B.1 and I.4 (pp. 18, 28), which should be reconciled with the generated-frame-feedback wording in Section 4.2 (pp. 6–7), rather than treated as absent. Finally, Table 3 already reports held-out predicate–object tests and Table 4 already removes the primitive planner/decomposition (p. 8). What remains insufficiently specified is the split by source episode across views and learned modules, and whether a matched-data comparison isolates short-horizon modeling from planner removal (Sections 2.2/3.1 and Appendix D/E, pp. 3–5, 20–22). This is an uncertainty about the reported protocol, not evidence that leakage occurred or that no compositional/decomposition experiment was performed.

Questions for authors
  1. Which assumption guarantees the local endpoint controllability and uniform trajectory tracking used in Theorem A.4? Can the authors provide a valid corrected statement, or withdraw the density and information-theoretic optimality claims?

  2. Which measured configuration yields the four-step, 12-FPS result, and which yields Tables 9–10? Please map the success-rate tables to these configurations, and reconcile the real-observation loop in the appendix with Section 4.2. Full action latency, including pose extraction, would determine whether the real-time control claim is supported.

  3. For Table 3's explicitly marked unseen combinations, what is held out from the video model versus the planner/grounder, and are views from the same source demonstration kept in one split? Table 4's planner-removal ablation is useful; does it also match video horizon, training data, and compute, or does it test the complete decomposition/planning module? This distinction would affect how strongly the results support the short-horizon data-efficiency claim.

Comments on the paper

1 comment

Dear Reviewers,

Please read the authors’ response and let us know whether it addresses your concerns. If any concerns remain, please briefly indicate which ones.

Thank you for your time and for your thoughtful reviews.

Best,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.