acceptodds
Under review as a conference paper at ICLR 2027

ExeWorldBench: Benchmarking Code-Based Executable 3D World Modeling

Abstract

Coding agents are increasingly evolving from general-purpose programming tools into a means for constructing and simulating executable worlds. Existing approaches have begun to explore this direction from two perspectives. The first takes explicit textual descriptions as input and turns them into executable world that exhibit the specified behaviors. The second begins with visual observations and seeks to infer the hidden physical properties, dynamics, and structural organization that explain the world. However, executable world modeling goes beyond either paradigm alone, requiring both world understanding and transferring that understanding into executable code-based construction. Yet, this integrated capability remains insufficiently evaluated. We introduce ExeWorldBench, the first benchmark for evaluating this capability across three complementary tasks.Mechanism Executability evaluates executable realization of individual mechanisms from visual observations. Compositional Executability extends this to multi-stage systems with spatial and causal dependencies. Intent-Driven Executable World Synthesis requires models to construct executable worlds from high-level goals and rules. Together, these tasks form a progressive evaluation hierarchy with increasing complexity and autonomy. Experiments on frontier models reveal substantial gaps in executable world modeling, underscoring the challenge of integrating world understanding with executable world construction.

Primary area:
datasets and benchmarks
Keywords:
Executable 3D World Modeling, Coding Benchmark

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_F3ijOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ExeWorldBench evaluates whether models can construct executable 3D worlds with coherent mechanical and causal structure, rather than merely producing runnable or visually plausible code. It covers mechanism reconstruction (T1), multi-stage causal composition (T2), and intent-driven synthesis (T3).

Critical strengths
  • Important benchmark direction that jointly tests physical understanding and executable realization.
  • Interesting empirical finding, as fixing code executability produces much smaller improvements in mechanical/causal correctness, showing that runnable code is not sufficient.
  • Well-designed task progression, where T1 -> T2 -> T3 increases causal complexity and autonomy.
Critical weaknesses
  1. Evaluation does not convincingly verify physical causality. The paper argues that behavioral correctness alone cannot establish that the underlying mechanism is correct. However, Visual Structure, Motion, ECM, and Goal Satisfaction are largely judged by a VLM from rendered frames. This leaves open whether the intended contacts, joints, constraints, and forces actually caused the behavior rather than scripted/keyframed shortcuts.
  2. Benchmark construction is strongly coupled to GPT-6 Astra. GPT-6 Astra participates in T1 filtering and validation, T2 generation and reference construction, as well as T3 task generation, while GPT-6 Astra systems are also evaluated. This creates a strong generator benchmark alignment concern. Results stratified by independently human-generated task provenance would help.
  3. Related work and novelty positioning should be strengthened. The discussion currently under-represents recent work arguing that visual fidelity alone is insufficient for evaluating world models. In particular: i) World-in-World (https://arxiv.org/abs/2510.18135) evaluates world models in closed-loop embodied interaction and finds that visual quality does not necessarily translate into task success; controllability and decision utility matter, ii) WorldModelBench (https://arxiv.org/abs/2502.20694) explicitly evaluates instruction following and physical-law adherence rather than generic video quality, and validates its judge against a large set of human annotations, iii) WorldScore (https://arxiv.org/abs/2504.00983) evaluates generated worlds along controllability, quality, and dynamics across 3D/4D and video-generation settings. These works are not equivalent to ExeWorldBench, but they make closely related arguments about moving beyond surface level visual quality. The paper should explain more precisely that its distinctive contribution is code-based executable mechanism construction and causal or mechanical fidelity, rather than broadly implying that prior world-model benchmarks mainly assess appearance.
Questions for authors

NA

Reviewer_KkjZOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Existing world modeling benchmarks focus on pure world understanding, such as question answering, or on behavioral correctness. Therefore, this paper emphasizes coherent executable mechanisms and proposes a benchmark for jointly evaluating two capabilities: world understanding and executable realization. The benchmark progressively evaluates three tasks, emphasizing execution, composition, and understanding.

Critical strengths
  1. The benchmark focuses on executable world modeling, which is valuable to the community. The authors also evaluate frontier models on the benchmark, demonstrating substantial experimental effort.
Critical weaknesses
  1. The paper lacks human evaluation experiments to validate the alignment between the VLM judge and human, which is my main concern.
  2. The benchmark primarily focuses on mechanical systems, with limited diversity in subjects and scenes, lacking broader world modeling scenarios such as indoor and outdoor environments and human-object interactions.
Questions for authors

Please see weaknesses.

Reviewer_4sQPOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ExeWorldBench evaluates whether models can build executable mechanical systems in Blender. Its tasks progress from reconstructing a mechanism from visual input to composing multi-stage systems and designing mechanisms from textual goals. The evaluation separates code execution from visual structure, motion, causal handoffs, and goal satisfaction. The experiments compare coding agents and API-based models, and examine code repair, motion references, and increasing chain length.

Critical strengths
  1. The task progression is well motivated. Building a working mechanical system requires more than recognizing objects or writing code that executes. The move from individual mechanisms to composed systems and goal-driven design provides a useful setting for studying this gap.

  2. The analysis distinguishes execution failures from errors in the resulting behavior. Reporting scores both with execution failures included and over executable outputs is helpful. The breakdown of goal conditions and the analysis of longer chains also offer more insight than a single overall ranking.

Critical weaknesses
  1. The evaluator has not been shown to measure the capability that makes this benchmark distinctive. Section 4.1 scores causal handoffs and goal conditions from sampled rendered frames. Yet a visually convincing animation can reproduce the expected sequence without implementing the required mechanical dependencies. The appendix specifies contact and event-order requirements, but the paper does not explain how these are verified or report agreement with human or simulation-state checks. This is particularly important because related work such as VisPhyWorld already evaluates executable reconstruction, and the revised PhysCodeBench includes direct physical checks. ExeWorldBench's contribution is most convincing as an evaluation of composed mechanisms and goal-driven design; the current scoring does not yet establish that distinction.

  2. The experimental protocol is not documented well enough to interpret the model comparisons. Coding agents and API-based models have different access to execution feedback, but tool and interaction budgets are not reported. The repair step is said to preserve the intended mechanism, without explaining how this restriction was checked. Exact task counts and the judge prompts and frame-sampling settings are also needed. The results can support a comparison between complete systems, but conclusions about the underlying models' physical reasoning are difficult to draw without these details.

Questions for authors
  1. How does the evaluator distinguish a required causal handoff from independently scheduled motions that look similar? Is there existing human or simulation-state validation of ECM and Goal Satisfaction?
  2. What are the tool and interaction budgets for each setting, and how was preservation of the intended mechanism during repair checked?
  3. Please provide the task counts, default visual inputs, and judge prompts and frame-sampling settings, including how tasks without reference videos are scored.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.