- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces WorldExam, a benchmark for evaluating video world models beyond visual quality and explicit control following. The main idea is to evaluate world reactivity: whether a model can infer plausible consequences that are implied by the scene but not explicitly stated in the input. The benchmark contains 1,474 cases across eight tasks and evaluates 23 camera-, action-, and language-driven models. It combines geometry-based metrics with checklist-based VLM evaluation. One interesting finding is that good control following does not necessarily imply good interaction or physical reasoning, and vice versa.
Critical strengths
-
A meaningful evaluation target. The paper goes beyond checking whether a model follows an instruction and asks whether the world reacts correctly to that action. I think this is an important capability that current world-model benchmarks do not fully capture.
-
Broad and solid experimental coverage. The evaluation covers 23 models and multiple control paradigms. I also appreciate that the authors use geometry-based metrics where possible instead of relying entirely on a VLM judge. The human-alignment study and reconstruction-backend comparison further strengthen the empirical validation and show that the authors have made a serious effort to investigate evaluator reliability.
Critical weaknesses
-
The VLM evaluator could use stronger validation. For several World Reactivity tasks, GPT-5.5 judges only 10 sampled frames. The reported human correlation is encouraging, but the human annotators also see the same 10 frames. This shows agreement with humans under the same observation setting, but does not fully show that 10 frames are enough to capture short interaction events in the full video.
-
Cross-interface comparison is not perfectly controlled. Camera, action, and language driven models receive different forms of control, and language models do not have explicit action durations. This makes some direct comparisons harder to interpret, especially when the paper draws conclusions across model paradigms.
Questions for authors
-
How sensitive are the checklist results to the number of sampled frames? Would using 20 frames or full-video human evaluation on a subset change the model rankings?
-
Could the authors provide a few representative failure cases of the GPT-5.5 evaluator, or report agreement statistics beyond correlation?
-
How robust are the cross-interface conclusions to different prompt formulations or temporal segmentation strategies for language-driven models?
4 comments
Response to Reviewer bDho (Part 2)
Q2: Could the authors provide representative failure cases of the GPT-5.5 evaluator, or agreement statistics beyond correlation?
Thank you for suggesting both quantitative agreement statistics and an analysis of evaluator failure cases. We report item-level agreement with human majority-vote labels, alongside correlation with full-video human assessments:
| Agreement with full-video human assessments | 10 frames | 20 frames | 40 frames |
|---|---|---|---|
| Spearman ρ | 0.8319 | 0.8169 | 0.8122 |
| Pearson r | 0.8292 | 0.8108 | 0.8086 |
| Item-level agreement | 87.29% | 84.84% | 85.31% |
We report the 10-frame item-level agreement in Section 5.4 and the results for all three frame budgets in Appendix F.5.
Regarding failure cases of the VLM evaluator, our evaluation prompt explicitly states: “use 1 only if the video clearly satisfies the item” and “use 0 if the item is contradicted, missing, ambiguous, partially satisfied, off-screen, or impossible to verify.” Consequently, when visual degradation makes a reaction difficult to judge, the evaluator tends to return False. We provide two examples in the supplementary material. In failure_case_1.mp4, nearby people make room for the protagonist, but occlusion and distorted figures cause the evaluator to mark the crowd-shifting item False. In failure_case_2.mp4, the evaluator recognizes the sign’s displacement after the cart’s impact, but warping and ghosting make it unclear whether its base and legs react first, leading to False for that specific item.
This evaluation policy is conservative when visual evidence is unclear: it avoids assigning high scores when occlusion or ambiguity prevents verification of the response. It also reflects our expectation that world models generate videos in which both the reactions and their causal relationships remain visually coherent and assessable.
Response to Reviewer bDho (Part 3)
Q3: How robust are the cross-interface conclusions to different prompt formulations or temporal segmentation strategies for language-driven models?
Thank you for suggesting these robustness checks. We examine prompt formulation and temporal segmentation separately.
(1) Prompt formulation.
We appreciate your suggestion to examine whether more explicit prompts improve control execution. We test three templates on 30 sampled cases, keeping the initial images and requested action sequences fixed. For a three-action Camera Control case, the templates are:
- Default: “The camera [action 1], then [action 2], then [action 3]. [Scene description]”
- Explicit order: “First, the camera [action 1]. Next, the camera [action 2]. Finally, the camera [action 3]. [Scene description]”
- Explicit timing: “During the first third of the video, the camera [action 1]. During the middle third of the video, the camera [action 2]. During the final third of the video, the camera [action 3]. [Scene description]”
We apply the three prompt templates only to language-driven models. Camera- and action-driven scores are obtained under their default settings on the same cases and are included for comparison. Neither explicit ordering nor explicit timing consistently improves performance across models and tasks. In this experiment, the two language-driven models score below the camera- and action-driven references under all three templates.
Camera Control
| Model | Interface | Default | Explicit order | Explicit timing |
|---|---|---|---|---|
| NeoVerse | Camera | 95.50 | — | — |
| InSpatio-World | Camera | 85.86 | — | — |
| WorldPlay | Action | 94.99 | — | — |
| LingBot-World-V2 | Action | 63.06 | — | — |
| Veo 3.1 | Language | 42.36 | 36.86 | 28.29 |
| Vidu Q3 | Language | 53.22 | 30.39 | 40.35 |
Subject Control
| Model | Interface | Default | Explicit order | Explicit timing |
|---|---|---|---|---|
| WorldPlay | Action | 55.31 | — | — |
| LingBot-World-V2 | Action | 72.22 | — | — |
| Veo 3.1 | Language | 20.29 | 13.40 | 28.11 |
| Vidu Q3 | Language | 36.11 | 43.02 | 36.56 |
(2) Temporal segmentation.
Thank you for highlighting the potential influence of automatic segmentation. For Camera and Subject Control, which involve multi-action sequences, we ask annotators to segment the language-driven models’ outputs by watching the full videos. We compare scores obtained using these human annotations with those using automatic segmentation. The scores vary modestly under human segmentation, while our main conclusions remain supported. We include this comparison in Table 7 and discuss the results in Section 5.4 of the revised manuscript.
| Model | Camera: Automatic | Camera: Human | Subject: Automatic | Subject: Human |
|---|---|---|---|---|
| Kling 2.5 | 50.18 | 54.77 | 28.40 | 30.62 |
| Veo 3.1 | 40.83 | 42.79 | 37.28 | 38.81 |
| Hailuo 2.3 | 63.29 | 65.11 | 36.49 | 38.25 |
| Wan 2.6 | 57.72 | 58.72 | 29.02 | 29.14 |
| Seedance 1.5 | 49.18 | 48.99 | 32.51 | 33.80 |
| Vidu Q3 | 42.75 | 43.05 | 27.67 | 31.56 |
| HappyHorse 1.0 | 58.29 | 58.83 | 33.11 | 32.25 |
We thank you again for your insightful comments and hope that these clarifications and additional experiments address your concerns.
[1] Shang, Yu, et al. “WorldArena: A Unified Benchmark for Evaluating Perception and Functional Utility of Embodied World Models.” 2026.
[2] Fu, Chaoyou, et al. “Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis.” CVPR, 2025.
Thank for your rebuttal. All of my concerns have been addressed. I will keep my score.
Response to Reviewer bDho (Part 1)
We sincerely thank you for recognizing the importance of evaluating world reactivity and the breadth of our experimental validation. Your constructive suggestions help us strengthen the full-video human evaluation and test sensitivity to temporal segmentation and prompt formulations.
We thank you and Reviewers TP7N and Ce4T for highlighting the potential blind spots of shared frame sampling. We use uniform sampling, as also adopted in benchmarks such as WorldArena [1] and Video-MME [2], with a 10-frame budget to balance temporal coverage and evaluation cost. To address your concern, annotators reassess the full, un-subsampled videos, and we compare VLM scores using 10, 20, and 40 uniformly sampled frames against these human assessments. The overall model ranking remains consistent with the original evaluation under the full-video human assessment.
The 10-frame scores correlate with full-video human judgments, and denser sampling does not increase correlation on this set. These results support the use of 10 frames for efficient evaluation. We update the human-alignment results in Section 5.4 (Table 5) and report the frame-budget analysis in Appendix F.5 (Tables 12–13). The pairwise Spearman correlation matrix below reports correlations between the 10-frame scores and those using 20 or 40 frames, suggesting broadly consistent results across frame budgets.
We thank you for highlighting this important distinction. We agree that differences in control interfaces should be considered when interpreting cross-paradigm comparisons. Accordingly, our reactivity tasks (Terrain Interaction, Object Interaction, Social Interaction, and Physical Reaction) all use a single atomic control to reduce cross-interface confounding associated with multi-action execution.
For Camera and Subject Control, language-driven prompts specify motion directions and their order without prescribing per-action durations. The evaluator allows action boundaries to follow the generated motion, making the timing requirements less restrictive than those for camera- and action-driven models. Nevertheless, language-driven models still obtain low scores on both tasks. These results suggest difficulties in executing the requested motions and their order, even without fixed per-action timing requirements.
For prompt formulation and evaluation-time temporal segmentation, which are specific to language-driven models in our protocol, we provide robustness analyses in our response to Q3.