- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Video2World, a benchmark for evaluating whether coding agents can reconstruct executable interactive simulations from embodied videos. It contains 222 instances from 189 videos across 39 task families. Agents must generate both the environment and robot behavior, and are evaluated on executability, geometry, dynamics, and task completion.
The experiments show that current agents usually produce executable simulations but remain unreliable at completing the demonstrated tasks. The paper also finds that geometric fidelity does not necessarily imply functional correctness.
Critical strengths
- Important and novel task. The benchmark integrates video understanding, physical reasoning, coding, simulation, and robot control.
- Comprehensive metrics. Separating geometry, dynamics, and functionality reveals failure modes that visual similarity alone would miss.
- Diverse benchmark. The dataset covers multiple sources, embodiments, simulators, and rigid, articulated, and deformable-object tasks.
Critical weaknesses
-
System comparisons are not fully controlled. The evaluated models use different agent frameworks, reasoning settings, stopping rules, costs, and runtime budgets. The results should therefore be presented strictly as comparisons of complete agent systems, not underlying models.
-
The human-assisted reference is not a controlled upper bound. It uses privileged annotations, source assets, evaluation feedback, and multiple refinement attempts. Its execution conditions also differ from those of coding agents. Since it achieves only 58.8% task success, results on the reference-success subset should be reported.
-
Human evaluation is relatively weak. It uses 12 internal raters, limited and unbalanced case coverage, and treats “both poor” identically to “equivalent” in the preference model.
After reading the rebuttal and the comments from other reviewers, most of my concerns were resolved. Overall, I think this paper is quite interesting and can provide some insights for the community.
Questions for authors
- Can environment errors be separated from controller errors through cross-execution?
Thank you for your detailed review and constructive feedback
Dear Reviewer gia6,
Thank you for your careful review and constructive suggestions. We have added the requested experiments and expanded the human-evaluation analysis. Our responses are below.
W1. Comparisons of complete agent systems
Response: Thank you for highlighting this. We have clarified that the main results compare complete coding-agent systems. Appendix D.2 adds a comparison of Fable, Opus, and DeepSeek using Claude Code, the same public harness, and a 180-minute construction budget on 60 instances spanning 20 task types.
The task-success ordering agrees with Table 2. Across three runs on the same fixed set of 20 instances, Fable and Opus achieve mean Task Success rates of 26.7% and 18.3%, respectively, each with a sample standard deviation of 2.9 percentage points.
W2. HAR and reference-success subset results
Response: Thank you for this suggestion. We have clarified HAR's role as an empirical reference and added the subset results in Appendix D.3. Its resources and execution conditions are documented in Appendix B.5.
The ordering is preserved on the HAR-success subset. Fable and Opus also complete approximately one-fifth of the instances where HAR fails.
W3. Human evaluation: raters, coverage, and ties
Response: Thank you for identifying these concerns. Appendix B.6 now addresses each point.
(i) Rater pool. The 12 internal raters include four paper authors. Across all 495 ways of removing four raters, the top-three set {HAR, Astra, Fable} is preserved in 493 combinations, and GLM remains lowest-rated in 494. Their internal ordering varies, consistent with overlapping intervals. HPR provides a complementary assessment of scene-and-interaction faithfulness.
(ii) Coverage and balance. The 243 valid judgments cover 81 case–system-pair comparisons across 39 cases, 19 task families, and seven video sources, with three judgments per comparison. Selection and pair assignments were fixed before rating, independently of automatic scores, and left/right presentation was randomized.
The comparisons cover 43 of 45 system pairs, forming a connected graph. Appendix B.6 reports source and task-family coverage and pairwise comparison counts.
(iii) "Both poor" and "equivalent." Both indicate no relative preference but differ in absolute quality. We report them separately and test four alternatives: excluding "both poor," using decisive judgments only, fitting a Davidson tie model, and treating "both poor" as a tie plus a loss for each candidate against a shared quality anchor.
The Davidson and quality-anchor models preserve the full ranking. Both exclusion analyses preserve the top-three set and swap two adjacent pairs (Kendall's τ = 0.91). Appendix B.6 reports these results and per-system "both poor" rates.
Q1. Separating environment and controller errors
Response: Appendix D.5 adds cross-execution experiments covering 447 agent–instance pairs. We exchange agent and HAR scenes and controllers, then check whether originally successful controllers remain successful after scene replacement.
Denominators count successes in the controller's original scene. FurnitureBench agent-controller transfers are also reported separately because its HAR reference uses closed-loop control.
In the can-lifting example, Fable's controller succeeds in the HAR scene, while the reverse combination fails, providing evidence of a scene-side issue. Appendix F.2 discusses this case and adds receiving-opening and cavity-depth measurements for bowl nesting.
Thank you again for your constructive feedback. We would be happy to clarify any remaining questions.
Best regards,
Authors of Submission 9774