- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
ExeWorldBench evaluates whether models can construct executable 3D worlds with coherent mechanical and causal structure, rather than merely producing runnable or visually plausible code. It covers mechanism reconstruction (T1), multi-stage causal composition (T2), and intent-driven synthesis (T3).
Critical strengths
- Important benchmark direction that jointly tests physical understanding and executable realization.
- Interesting empirical finding, as fixing code executability produces much smaller improvements in mechanical/causal correctness, showing that runnable code is not sufficient.
- Well-designed task progression, where T1 -> T2 -> T3 increases causal complexity and autonomy.
Critical weaknesses
- Evaluation does not convincingly verify physical causality. The paper argues that behavioral correctness alone cannot establish that the underlying mechanism is correct. However, Visual Structure, Motion, ECM, and Goal Satisfaction are largely judged by a VLM from rendered frames. This leaves open whether the intended contacts, joints, constraints, and forces actually caused the behavior rather than scripted/keyframed shortcuts.
- Benchmark construction is strongly coupled to GPT-6 Astra. GPT-6 Astra participates in T1 filtering and validation, T2 generation and reference construction, as well as T3 task generation, while GPT-6 Astra systems are also evaluated. This creates a strong generator benchmark alignment concern. Results stratified by independently human-generated task provenance would help.
- Related work and novelty positioning should be strengthened. The discussion currently under-represents recent work arguing that visual fidelity alone is insufficient for evaluating world models. In particular: i) World-in-World (https://arxiv.org/abs/2510.18135) evaluates world models in closed-loop embodied interaction and finds that visual quality does not necessarily translate into task success; controllability and decision utility matter, ii) WorldModelBench (https://arxiv.org/abs/2502.20694) explicitly evaluates instruction following and physical-law adherence rather than generic video quality, and validates its judge against a large set of human annotations, iii) WorldScore (https://arxiv.org/abs/2504.00983) evaluates generated worlds along controllability, quality, and dynamics across 3D/4D and video-generation settings. These works are not equivalent to ExeWorldBench, but they make closely related arguments about moving beyond surface level visual quality. The paper should explain more precisely that its distinctive contribution is code-based executable mechanism construction and causal or mechanical fidelity, rather than broadly implying that prior world-model benchmarks mainly assess appearance.
Questions for authors
NA