- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
VIDEAS aims to distill explicit action semantics—preconditions and effects—from demonstration videos into compact world models. Starting from AgiBot-World, the authors filter and deduplicate episodes, use a VLM to derive structured descriptions, and apply prior-guided simulation to generate action traces and counterfactual negatives with violated prerequisites. The resulting data are used to fine-tune Llama3.1-8B, Qwen3-8B, and Qwen3.5-9B. The paper evaluates on BTSIMBENCH, LogicEnvEval, and an accessible subset of WorldPrediction-WM. It reports large gains on the first two symbolic/simulation benchmarks, while results on WorldPrediction-WM are mixed and do not uniformly surpass stronger VLM baselines.
Critical strengths
- The paper turns demonstration videos into structured precondition/effect supervision, making the intended action semantics explicit and enabling targeted evaluation of whether a model anticipates action prerequisites.
- It includes counterfactual negative trajectories and an ablation showing a substantial loss in counterfactual accuracy when these examples are removed. This is a useful test of the proposed data design rather than only a model-size comparison.
- The gains on BTSIMBENCH and LogicEnvEval are large across several backbone families, and the paper also evaluates on a separate world-prediction benchmark rather than relying exclusively on its own training distribution.
Critical weaknesses
- Only 135 of approximately 34K generated training examples are manually audited. The paper says the items were randomly sampled and labels were assigned by majority vote of three annotators, but gives no sampling seed or stratification, confidence intervals, or numerical inter-annotator agreement statistic; “high agreement” is not reproducible as reported. The audit supports the quality of this small sample, not corpus-wide correctness across the long tail of tasks and action types.
- The data generator and verifier both rely on model-generated symbolic descriptions and prior-guided simulation. The paper does not fully establish how independently the verification catches errors shared by the VLM, the action schema, and the simulator, or how often generated trajectories are valid under physical execution. This makes it difficult to distinguish semantic grounding from improved consistency on the constructed symbolic interface.
- The strongest results are on small simulation-oriented evaluations. On WorldPrediction-WM, VIDEAS-WM-9B scores 58.33, only 1.88 points above Qwen3-VL-32B, while the Socratic VIDEAS-WM-Q8B score of 63.42 is only 0.63 above DeepSeek-V4-Pro. More seriously, Table 2’s displayed percentages imply inconsistent denominators: VIDEAS-WM-9B is consistent with 636 items while base Qwen3-8B and Llama3.1-8B are consistent with 640 items. Appendix C.3 and Table 5 instead say all models use the same 637-item subset, with 193 EPIC and 90 CrossTask items. Until per-row counts are reconciled, the cross-model comparison is difficult to audit. The current evidence does not establish robust world-model or embodied physical understanding, and no robot execution is reported.
- The evaluation setup leaves smaller but consequential reporting details unclear. Table 1 reports both Delivery and Avg but does not define Avg; the displayed values appear to average the simulation-accuracy categories while excluding Delivery. The table abbreviations CFactuals, Unreach, and LBranch also differ from the prose labels Counterfactuals, Unreachable, and LackBranch. Appendix B reports batch size 4 and four GPUs without saying whether 4 is per GPU or global, leaving the effective batch size ambiguous.
Questions for authors
See weakness