acceptodds
Under review as a conference paper at ICLR 2027

VIDEAS: Distilling Explicit Action Semantics from Demonstration Videos for World Models via Prior-Guided Simulation

Abstract

World models learn internal representations of environment dynamics to predict future states, enabling agents to optimize action plans without physical interactions. However, developing world models that genuinely internalize underlying causal physical laws to explicitly reason about action preconditions and subsequent state transitions remains an open challenge. In this paper, we propose VIDEAS, a data distillation framework that transforms continuous physical dynamics from operational videos into explicit action semantics for foundation models. Specifically, it deconstructs visual demonstrations into discrete action trajectories and utilizes advanced vision-language models (VLMs) to extract structured knowledge encapsulating action preconditions and effects. To ensure physical consistency, we introduce a prior-guided trajectory simulation mechanism grounded within a text-based environment to rigorously validate the extracted knowledge. Notably, we incorporate negative trajectories to enrich knowledge completeness and enhance data diversity to mitigate cognitive bias. Furthermore, we present VIDEAS-WM, an 8B/9B-parameter suite of language-based world models trained on 34K high-quality samples derived from AgiBot-World dataset. Extensive experiments demonstrate that VIDEAS-WM establishes state-of-the-art performance in high-level embodied action semantic reasoning, exhibiting profound physical understanding and robust generalization across unseen scenarios.

Primary area:
applications to robotics, autonomy, planning
Keywords:
world model, embodied ai

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_YNnZOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

VIDEAS aims to distill explicit action semantics—preconditions and effects—from demonstration videos into compact world models. Starting from AgiBot-World, the authors filter and deduplicate episodes, use a VLM to derive structured descriptions, and apply prior-guided simulation to generate action traces and counterfactual negatives with violated prerequisites. The resulting data are used to fine-tune Llama3.1-8B, Qwen3-8B, and Qwen3.5-9B. The paper evaluates on BTSIMBENCH, LogicEnvEval, and an accessible subset of WorldPrediction-WM. It reports large gains on the first two symbolic/simulation benchmarks, while results on WorldPrediction-WM are mixed and do not uniformly surpass stronger VLM baselines.

Critical strengths
  • The paper turns demonstration videos into structured precondition/effect supervision, making the intended action semantics explicit and enabling targeted evaluation of whether a model anticipates action prerequisites.
  • It includes counterfactual negative trajectories and an ablation showing a substantial loss in counterfactual accuracy when these examples are removed. This is a useful test of the proposed data design rather than only a model-size comparison.
  • The gains on BTSIMBENCH and LogicEnvEval are large across several backbone families, and the paper also evaluates on a separate world-prediction benchmark rather than relying exclusively on its own training distribution.
Critical weaknesses
  • Only 135 of approximately 34K generated training examples are manually audited. The paper says the items were randomly sampled and labels were assigned by majority vote of three annotators, but gives no sampling seed or stratification, confidence intervals, or numerical inter-annotator agreement statistic; “high agreement” is not reproducible as reported. The audit supports the quality of this small sample, not corpus-wide correctness across the long tail of tasks and action types.
  • The data generator and verifier both rely on model-generated symbolic descriptions and prior-guided simulation. The paper does not fully establish how independently the verification catches errors shared by the VLM, the action schema, and the simulator, or how often generated trajectories are valid under physical execution. This makes it difficult to distinguish semantic grounding from improved consistency on the constructed symbolic interface.
  • The strongest results are on small simulation-oriented evaluations. On WorldPrediction-WM, VIDEAS-WM-9B scores 58.33, only 1.88 points above Qwen3-VL-32B, while the Socratic VIDEAS-WM-Q8B score of 63.42 is only 0.63 above DeepSeek-V4-Pro. More seriously, Table 2’s displayed percentages imply inconsistent denominators: VIDEAS-WM-9B is consistent with 636 items while base Qwen3-8B and Llama3.1-8B are consistent with 640 items. Appendix C.3 and Table 5 instead say all models use the same 637-item subset, with 193 EPIC and 90 CrossTask items. Until per-row counts are reconciled, the cross-model comparison is difficult to audit. The current evidence does not establish robust world-model or embodied physical understanding, and no robot execution is reported.
  • The evaluation setup leaves smaller but consequential reporting details unclear. Table 1 reports both Delivery and Avg but does not define Avg; the displayed values appear to average the simulation-accuracy categories while excluding Delivery. The table abbreviations CFactuals, Unreach, and LBranch also differ from the prose labels Counterfactuals, Unreachable, and LackBranch. Appendix B reports batch size 4 and four GPUs without saying whether 4 is per GPU or global, leaving the effective batch size ambiguous.
Questions for authors

See weakness

Reviewer_zqgPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

VIDEAS distills action preconditions, their satisfaction, and effects from demonstration videos through prior-guided trajectory simulation and dependency-aware negative trajectories. The resulting 34K examples are used to fine-tune 8B/9B models. Evaluation on BTSIMBENCH, LogicEnvEval, and WorldPrediction-WM shows substantial improvements in text-based behavior simulation and additional gains in cross-scenario action reasoning.

Critical strengths
  • Well-motivated supervision. Explicitly modeling action preconditions addresses an important failure mode: predicting successful transitions despite unmet requirements. Dependency-aware negative trajectories provide targeted supervision for recognizing such failures.

  • Strong empirical improvements. Gains are consistent across three backbones. The negative-trajectory ablation is particularly compelling: Qwen3.5-9B’s counterfactual accuracy increases from 48% to 96%, supporting the practical importance of negative examples.

Critical weaknesses
  • The claimed physical understanding is not fully supported. The method checks whether simulated actions succeed or fail as expected, but this does not guarantee that the learned preconditions and effects are physically correct. The experiments mainly evaluate semantic reasoning rather than real physical interactions.
  • Unvalidated downstream utility. The experiments demonstrate improved semantic simulation and action reasoning, but do not evaluate integration with a planner or robot policy in closed-loop execution. It therefore remains unclear whether these gains translate into higher robotic task success rates.
Questions for authors
  • Were the synthesized failure points independently checked, rather than only validated against the LLM-generated dependencies?

  • Have you evaluated whether using VIDEAS-WM for action validation or planning improves closed-loop robotic task success over the same system without it?

Reviewer_8cLwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

VIDEAS distills action preconditions and effects from demonstration videos using a VLM teacher. It filters generated annotations through trajectory simulation constrained by successful demonstrations and synthesized failure cases. The resulting 34K examples are used to fine-tune 8B/9B models for action-semantic reasoning.

Critical strengths

Explicitly modeling action preconditions addresses a meaningful limitation of unconditional state-transition prediction.

Improvements across three model backbones and the negative-trajectory ablation provide evidence that the curated data benefits the evaluated tasks.

Critical weaknesses

Limited evaluation scale and coverage. The two primary benchmarks contain only 75 and 100 policies, respectively, with 25 examples per category. This provides limited evidence for broad generalization. The additional 637-example WorldPrediction evaluation is useful, but tests action identification rather than downstream planning effectiveness.

Idealized settings limit practical conclusions. The primary evaluations provide structured, fully specified textual states, abstracting away perception errors, missing information, and continuous physical constraints. This is reasonable for high-level semantic modeling, but the paper does not sufficiently demonstrate that the gains translate into more reliable embodied planning. Evaluations with state noise, incomplete observations, or independently assessed planning outcomes would strengthen the contribution.

The contribution of video grounding is not isolated. Although the paper includes naive-distillation and negative-trajectory ablations, it lacks a matched text-only distillation baseline using the same teacher, scene descriptions, and action sequences. Consequently, it remains unclear whether visual evidence contributes substantially beyond the teacher’s existing knowledge and task-specific fine-tuning.

Outcome consistency does not establish semantic correctness. Requiring simulated trajectories to match expected success or failure outcomes can filter inconsistencies, but incomplete preconditions or incorrect intermediate transitions may still satisfy these constraints. Moreover, the negative execution priors themselves depend on LLM-inferred dependencies. The 135-instance human audit is encouraging, but does not adequately establish the completeness and reliability of the extracted rules across counterfactual situations.

Questions for authors

Please see the weakness section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.