acceptodds
Under review as a conference paper at ICLR 2027

Benchmarking Coding Agents for Interactive World Modeling from Embodied Videos

11 views

Abstract

Building interactive simulators from real-world observations is a promising way to scale embodied data, but current pipelines still rely heavily on manual environment construction and calibration. We study whether frontier foundation models and coding agents can automate this process end to end. We formulate autonomous video-to-simulation as a software engineering task in which an agent observes an embodied video, constructs the corresponding simulated environment and robot behavior, and iteratively refines the result through execution feedback. To evaluate this capability, we introduce Video2World, a benchmark comprising 222 reconstruction instances derived from 189 robot and human demonstration videos. Video2World measures reconstructed worlds along geometric fidelity, dynamic fidelity, and functional correctness, capturing spatial perception, physical reasoning, and executable interaction. Evaluating 9 frontier coding-agent systems reveals a sharp improvement in Task success beginning with Claude Opus 5, rising from below 5% to over 15%, while substantial gaps to human-assisted reconstruction remain. We further find that worlds that look better could work worse: better visual fidelity does not always lead to higher task success. This echoes the broader gap between perceptual realism and factual correctness observed in generative models. Our data and code will be publicly released.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Coding Agent, Real2Sim2Real, Embodied AI

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_gia6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Video2World, a benchmark for evaluating whether coding agents can reconstruct executable interactive simulations from embodied videos. It contains 222 instances from 189 videos across 39 task families. Agents must generate both the environment and robot behavior, and are evaluated on executability, geometry, dynamics, and task completion.

The experiments show that current agents usually produce executable simulations but remain unreliable at completing the demonstrated tasks. The paper also finds that geometric fidelity does not necessarily imply functional correctness.

Critical strengths
  1. Important and novel task. The benchmark integrates video understanding, physical reasoning, coding, simulation, and robot control.
  2. Comprehensive metrics. Separating geometry, dynamics, and functionality reveals failure modes that visual similarity alone would miss.
  3. Diverse benchmark. The dataset covers multiple sources, embodiments, simulators, and rigid, articulated, and deformable-object tasks.
Critical weaknesses
  1. System comparisons are not fully controlled. The evaluated models use different agent frameworks, reasoning settings, stopping rules, costs, and runtime budgets. The results should therefore be presented strictly as comparisons of complete agent systems, not underlying models.

  2. The human-assisted reference is not a controlled upper bound. It uses privileged annotations, source assets, evaluation feedback, and multiple refinement attempts. Its execution conditions also differ from those of coding agents. Since it achieves only 58.8% task success, results on the reference-success subset should be reported.

  3. Human evaluation is relatively weak. It uses 12 internal raters, limited and unbalanced case coverage, and treats “both poor” identically to “equivalent” in the preference model.


After reading the rebuttal and the comments from other reviewers, most of my concerns were resolved. Overall, I think this paper is quite interesting and can provide some insights for the community.

Questions for authors
  1. Can environment errors be separated from controller errors through cross-execution?
1 comment
AuthorsOpenReview ↗

Thank you for your detailed review and constructive feedback

Dear Reviewer gia6,

Thank you for your careful review and constructive suggestions. We have added the requested experiments and expanded the human-evaluation analysis. Our responses are below.

W1. Comparisons of complete agent systems

"The results should therefore be presented strictly as comparisons of complete agent systems, not underlying models."

Response: Thank you for highlighting this. We have clarified that the main results compare complete coding-agent systems. Appendix D.2 adds a comparison of Fable, Opus, and DeepSeek using Claude Code, the same public harness, and a 180-minute construction budget on 60 instances spanning 20 task types.

SystemScored instancesTask SuccessV2WScore
Fable 5.15933.9%45.61
Claude Opus 55815.5%34.02
DeepSeek V4.1 Flash596.8%23.74

The task-success ordering agrees with Table 2. Across three runs on the same fixed set of 20 instances, Fable and Opus achieve mean Task Success rates of 26.7% and 18.3%, respectively, each with a sample standard deviation of 2.9 percentage points.

W2. HAR and reference-success subset results

"The human-assisted reference is not a controlled upper bound."

"Since it achieves only 58.8% task success, results on the reference-success subset should be reported."

Response: Thank you for this suggestion. We have clarified HAR's role as an empirical reference and added the subset results in Appendix D.3. Its resources and execution conditions are documented in Appendix B.5.

SystemTask Success on HAR-success subsetTask Success on HAR-failure subset
Fable 5.129.9%20.3%
Claude Opus 521.2%21.2%
GPT-6 Astra9.4%5.9%

The ordering is preserved on the HAR-success subset. Fable and Opus also complete approximately one-fifth of the instances where HAR fails.

W3. Human evaluation: raters, coverage, and ties

"It uses 12 internal raters, limited and unbalanced case coverage, and treats "both poor" identically to "equivalent" in the preference model."

Response: Thank you for identifying these concerns. Appendix B.6 now addresses each point.

(i) Rater pool. The 12 internal raters include four paper authors. Across all 495 ways of removing four raters, the top-three set {HAR, Astra, Fable} is preserved in 493 combinations, and GLM remains lowest-rated in 494. Their internal ordering varies, consistent with overlapping intervals. HPR provides a complementary assessment of scene-and-interaction faithfulness.

(ii) Coverage and balance. The 243 valid judgments cover 81 case–system-pair comparisons across 39 cases, 19 task families, and seven video sources, with three judgments per comparison. Selection and pair assignments were fixed before rating, independently of automatic scores, and left/right presentation was randomized.

The comparisons cover 43 of 45 system pairs, forming a connected graph. Appendix B.6 reports source and task-family coverage and pairwise comparison counts.

(iii) "Both poor" and "equivalent." Both indicate no relative preference but differ in absolute quality. We report them separately and test four alternatives: excluding "both poor," using decisive judgments only, fitting a Davidson tie model, and treating "both poor" as a tie plus a loss for each candidate against a shared quality anchor.

The Davidson and quality-anchor models preserve the full ranking. Both exclusion analyses preserve the top-three set and swap two adjacent pairs (Kendall's τ = 0.91). Appendix B.6 reports these results and per-system "both poor" rates.

Q1. Separating environment and controller errors

"Can environment errors be separated from controller errors through cross-execution?"

Response: Appendix D.5 adds cross-execution experiments covering 447 agent–instance pairs. We exchange agent and HAR scenes and controllers, then check whether originally successful controllers remain successful after scene replacement.

SystemAgent controller in HAR sceneHAR controller in agent scene
Fable 5.16/423/41
GPT-6 Astra1/161/41
Claude Opus 50/312/39

Denominators count successes in the controller's original scene. FurnitureBench agent-controller transfers are also reported separately because its HAR reference uses closed-loop control.

In the can-lifting example, Fable's controller succeeds in the HAR scene, while the reverse combination fails, providing evidence of a scene-side issue. Appendix F.2 discusses this case and adds receiving-opening and cavity-depth measurements for bowl nesting.

Thank you again for your constructive feedback. We would be happy to clarify any remaining questions.

Best regards,
Authors of Submission 9774

Reviewer_2uuNOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Video2World, a benchmark for evaluating whether coding agents can reconstruct interactive simulated worlds from embodied videos. The benchmark contains 222 reconstruction instances from 189 videos and evaluates nine coding-agent systems across geometry, dynamics, and task functionality. I find the problem interesting and timely. A particularly useful aspect is that the paper evaluates whether the reconstructed world actually supports the demonstrated interaction, rather than only measuring visual or geometric similarity. The experiments also show that better geometry does not always translate into better task success.

Critical strengths
  1. Meaningful end-to-end evaluation. The paper goes beyond static scene reconstruction and evaluates whether the reconstructed environment and robot behavior actually work together under physical execution. I think this is the most important contribution.

  2. Strong benchmark construction and analysis. The benchmark covers diverse videos, tasks, embodiments, and simulators, and the paper provides detailed evaluation metrics and useful failure analysis. The distinction between geometric fidelity and functional correctness is especially informative.

Critical weaknesses
  1. The comparison mixes model capability with agent-framework differences. Most models are evaluated through OpenCode, while GPT-6 Astra and Fable-5.1 use different native coding-agent frameworks. The paper itself notes that these results should be interpreted as comparisons of complete agent systems rather than isolated backbone models. This makes some of the model-to-model ranking difficult to interpret. I would like to see either a small controlled comparison under the same scaffold or repeated runs on a subset of tasks.

  2. V2WScore could use a sensitivity analysis. The final score combines functionality, geometry, and dynamics with fixed normalization scales and equal weighting. It would be useful to show whether the overall ranking remains stable under reasonable changes to these weights or thresholds.

Questions for authors
  1. How much of the performance difference comes from the underlying model versus the coding-agent framework? Could you provide a controlled experiment on a subset of models/tasks?

  2. How stable is V2WScore under different component weights or normalization thresholds?

  3. Since each system only gets one construction attempt per instance, do you have any estimate of run-to-run variance for the stronger agents?

1 comment
AuthorsOpenReview ↗

Thank you for your thoughtful feedback and helpful suggestions

Dear Reviewer 2uuN,

Thank you for your thoughtful review and for recognizing the value of evaluating executable interaction alongside geometric fidelity. Following your suggestions, we have added a common-framework comparison, a scoring sensitivity analysis, and repeated-run results. Please find our responses below.

Q1/W1. Model and coding-framework effects

"How much of the performance difference comes from the underlying model versus the coding-agent framework? Could you provide a controlled experiment on a subset of models/tasks?"

Response: Thank you for this suggestion. We have added a comparison with the coding framework held fixed. Fable, Opus, and DeepSeek are evaluated using Claude Code, the same public harness, and a 180-minute construction budget on 60 instances spanning 20 task types.

SystemScored instancesTask SuccessV2WScore
Fable 5.15933.9%45.61
Claude Opus 55815.5%34.02
DeepSeek V4.1 Flash596.8%23.74

The task-success ordering agrees with Table 2. Among jointly scored instances, 12 succeed only with Fable and one succeeds only with Opus. Appendix D.2 reports the full comparison. We continue to present the main benchmark results as comparisons of complete coding-agent systems.

Q2/W2. Sensitivity of V2WScore

"How stable is V2WScore under different component weights or normalization thresholds?"

Response: We have added both analyses in Appendix D.4.

For component weights, we test all 66 nonnegative weight triples summing to one on a grid with step size 0.1. The top-three set remains {Fable, Astra, Opus} in every setting. Fable ranks first in 56 settings; Astra ranks first only when the functionality weight is at most 0.1.

For normalization, we multiply the geometry and dynamics constants by 0.5, 0.75, 1.5, and 2, varying them separately and jointly. All tested settings preserve the complete ranking. Including Build Rate as a fourth equally weighted component also leaves the ranking unchanged.

We additionally test 20 settings for task-success and progress tolerances. This analysis covers 1,200 execution records across 18 task types; records outside this coverage retain their recorded scores. The leading Task Success ordering remains Fable > Opus > Astra, and the maximum V2WScore change is 1.6 points.

Q3/W1. Run-to-run variation

"Since each system only gets one construction attempt per instance, do you have any estimate of run-to-run variance for the stronger agents?"

Response: Yes. We evaluate both Fable and Opus on the same fixed set of 20 instances using the common framework described above. The subset contains one instance from each of 20 task types, with three construction runs per model on each instance.

SystemRun 1Run 2Run 3Mean ± sample SD
Fable 5.125.0%25.0%30.0%26.7% ± 2.9 pp
Claude Opus 520.0%15.0%20.0%18.3% ± 2.9 pp

Across the three runs, mean Task Success is 26.7% for Fable and 18.3% for Opus, with a sample standard deviation of 2.9 percentage points for each system. We report these results in Appendix D.2.

Thank you again for your helpful suggestions. We hope these additional experiments address your questions about framework effects, scoring sensitivity, and run-to-run variation.

Best regards,
Authors of Submission 9774

Reviewer_3JXxOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Video2World, a benchmark that frames video-to-simulation as an end-to-end coding task: given an RGB embodied video and a target simulator/robot, a coding agent must build the scene and a robot control sequence, and refine them using rollouts of its own reconstruction. The benchmark contains 222 instances from robot, egocentric and third-person human demonstrations across SAPIEN, Isaac Sim and MuJoCo, with hidden object-centric annotations used to measure geometric fidelity, dynamic fidelity, and task success/progress, summarized as V2WScore.

Critical strengths
  1. The task is timely and well posed. Object-centric annotations allow human and robot videos to be evaluated under different target embodiments, and the rule that objects can only move through robot actuation forces physically realized interactions rather than visually plausible scenes. This is the main reason I lean towards acceptance.

  2. The evaluation is careful and transparent: geometry, dynamics, and functionality are reported separately, the information boundary and budgets are clearly specified, and the case studies show concretely how geometric fidelity and task success can disagree.

Critical weaknesses
  1. Task Success is measured on the joint execution of the agent's scene and its own controls, so a failure cannot be attributed to the reconstructed world or to the behavior. This also makes the main finding hard to interpret: in Figures A9, A10 and A13 the reconstructions with the lowest Scene CD fail, but it is unclear whether the scene (e.g., collision geometry not captured by CD) or the controls are responsible.

  2. The comparisons in Table 2 are not controlled. HAR uses CAD, calibration, robot logs, simulator states and evaluation scores, so the gap to HAR is not a gap to human reconstruction from RGB. Among agents, the top two systems run in their native harnesses under a different schedule from the OpenCode runs, and each configuration has a single attempt per instance, so differences such as Fable 5.1 vs. Claude Opus 5 (25.5% vs. 16.0%) mix model, scaffold, and run-to-run variance.

Questions for authors
  1. (Clarification) Appendix C mentions fixed-reference-action replays as separate diagnostics. Could the authors report cross-execution results, i.e., agent controls executed in the annotated scene and reference controls executed in the agent's scene, to separate reconstruction errors from behavior errors?

  2. (Clarification) Could the authors run the strongest systems under a common harness (e.g., Claude Opus 5 under Claude Code, or Fable 5.1 under OpenCode) and report variance over repeated runs on a subset?

  3. (Clarification) The evaluation relies on hidden annotations. Will an evaluation server be provided, and how will the in-house household subset be released?

If the authors adequately address my concerns, I will consider raising my rating.

3 comments
AuthorsOpenReview ↗

Thank you for your careful review and constructive suggestions

Dear Reviewer 3JXx,

Thank you for your careful review and thoughtful suggestions. We appreciate your recognition of the task formulation and the emphasis on physically realized interaction. Following your comments, we have added cross-execution experiments, a common-framework comparison, and repeated-run results. Our point-by-point responses are below.

Q1/W1. Separating scene and controller errors

"Could the authors report cross-execution results, i.e., agent controls executed in the annotated scene and reference controls executed in the agent's scene, to separate reconstruction errors from behavior errors?"

Response: Thank you for this suggestion. We have added cross-execution experiments covering 447 agent–instance pairs in Appendix D.5. We exchange scenes and controllers between agent reconstructions and HAR references, then check whether controllers that succeed in their original scenes remain successful after scene replacement.

SystemAgent controller in HAR sceneHAR controller in agent scene
Fable 5.16/423/41
GPT-6 Astra1/161/41
Claude Opus 50/312/39

Each denominator counts successful executions in the controller's original scene. Appendix D.5 also reports FurnitureBench agent-controller transfers separately, as its HAR reference uses closed-loop control.

We connect these experiments to the case analysis in Appendix F.2. In the can-lifting example, Fable's controller succeeds in the HAR scene, while the HAR controller fails in Fable's scene. Changing the scene therefore enables the same agent controller to complete the task, providing evidence of a scene-side issue in this example.

For bowl nesting, we added measurements showing that the lowest-CD reconstruction has a narrower receiving opening and shallower cavity than the reference. We also checked the collision geometry you mentioned: replacing visual geometry with collision geometry in the evaluation changes Scene CD by at most 0.1 cm in the examined cases.

W2. Interpreting the HAR comparison

"HAR uses CAD, calibration, robot logs, simulator states and evaluation scores, so the gap to HAR is not a gap to human reconstruction from RGB."

Response: Thank you for highlighting this distinction. We have clarified in the main text that HAR is an assisted reference constructed with the additional resources documented in Appendix B.5. Its comparison with coding agents does not measure human reconstruction from RGB alone.

We also added agent results on the HAR-success and HAR-failure subsets in Appendix D.3:

SystemTask Success on HAR-success subsetTask Success on HAR-failure subset
Fable 5.129.9%20.3%
Claude Opus 521.2%21.2%
GPT-6 Astra9.4%5.9%

These results show agent performance on cases where HAR succeeds, as well as cases that agents solve despite HAR failing.

Q2/W2. Common-framework comparison and run-to-run variation

"Could the authors run the strongest systems under a common harness [...] and report variance over repeated runs on a subset?"

Response: Thank you for pointing this out. We have added both experiments in Appendix D.2. For the common-framework comparison, we evaluate Fable, Opus, and DeepSeek using Claude Code, the same public harness, and a 180-minute construction budget on 60 instances spanning 20 task types.

SystemScored instancesTask SuccessV2WScore
Fable 5.15933.9%45.61
Claude Opus 55815.5%34.02
DeepSeek V4.1 Flash596.8%23.74

The task-success ordering agrees with Table 2. Among jointly scored instances, 12 succeed only with Fable and one succeeds only with Opus.

For repeated runs, we evaluate both Fable and Opus on the same fixed set of 20 instances, with one instance per task type and three construction runs per model on each instance.

SystemRun 1Run 2Run 3Mean ± sample SD
Fable 5.125.0%25.0%30.0%26.7% ± 2.9 pp
Claude Opus 520.0%15.0%20.0%18.3% ± 2.9 pp

The table reports Task Success for each run and the mean and sample standard deviation across the three runs. These results are included in Appendix D.2.

Q3. Evaluation access and household data release

"Will an evaluation server be provided, and how will the in-house household subset be released?"

Response: Yes. The evaluation service will be available, and the evaluation code will be open-sourced. The public release includes the evaluation annotations and the complete in-house household subset, with its videos, scene assets, and annotations. Evaluation annotations are excluded from the coding agents' inputs during benchmark execution.

Thank you again for your constructive suggestions. We hope the added experiments and clarifications address your concerns, and we would be happy to answer any remaining questions.

Best regards,
Authors of Submission 9774

All of my concerns have been addressed, so I’ve raised my rating to Clear Accept. Good luck!

AuthorsOpenReview ↗

We sincerely appreciate your time and support

Thank you for your time and constructive feedback. Your suggestions helped us improve the paper. We appreciate your support and kind wishes!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.