acceptodds
Under review as a conference paper at ICLR 2027

WorldExam: Benchmarking World Models from Apparent Appearance to Inherent Reactivity

Abstract

Controllable video generation models are increasingly being developed as world models. Accordingly, evaluating them in this role extends beyond the *apparent appearance* of generated videos to the *inherent reactivity* of the worlds they depict: the ability to infer from the scene state how the world should react and to generate plausible consequences not explicitly described in the input. Yet existing benchmarks mainly assess visual quality or explicit instruction fulfillment by checking whether requested actions and interaction outcomes are realized, leaving inherent reactivity underexamined. We introduce **WorldExam**, a hierarchical diagnostic benchmark spanning four levels: Visual Quality, Control Adherence, Spatial Consistency, and World Reactivity. It comprises 1,474 cases across eight dedicated tasks and supports unified evaluation of camera-, action-, and language-driven model paradigms. The World Reactivity level evaluates scene-conditioned reactions with expected consequences left unstated and goal-directed behaviors with execution steps unspecified. Evaluation of 23 models shows that neither high visual quality nor strong control adherence guarantees inherent reactivity, while models with stronger reactivity can exhibit less reliable control. These findings suggest that progress requires combining these strengths, so that models both follow controls and generate plausible consequences implied by the scene.

Primary area:
datasets and benchmarks
Keywords:
Video World Model, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_bDhoOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces WorldExam, a benchmark for evaluating video world models beyond visual quality and explicit control following. The main idea is to evaluate world reactivity: whether a model can infer plausible consequences that are implied by the scene but not explicitly stated in the input. The benchmark contains 1,474 cases across eight tasks and evaluates 23 camera-, action-, and language-driven models. It combines geometry-based metrics with checklist-based VLM evaluation. One interesting finding is that good control following does not necessarily imply good interaction or physical reasoning, and vice versa.

Critical strengths
  1. A meaningful evaluation target. The paper goes beyond checking whether a model follows an instruction and asks whether the world reacts correctly to that action. I think this is an important capability that current world-model benchmarks do not fully capture.

  2. Broad and solid experimental coverage. The evaluation covers 23 models and multiple control paradigms. I also appreciate that the authors use geometry-based metrics where possible instead of relying entirely on a VLM judge. The human-alignment study and reconstruction-backend comparison further strengthen the empirical validation and show that the authors have made a serious effort to investigate evaluator reliability.

Critical weaknesses
  1. The VLM evaluator could use stronger validation. For several World Reactivity tasks, GPT-5.5 judges only 10 sampled frames. The reported human correlation is encouraging, but the human annotators also see the same 10 frames. This shows agreement with humans under the same observation setting, but does not fully show that 10 frames are enough to capture short interaction events in the full video.

  2. Cross-interface comparison is not perfectly controlled. Camera, action, and language driven models receive different forms of control, and language models do not have explicit action durations. This makes some direct comparisons harder to interpret, especially when the paper draws conclusions across model paradigms.

Questions for authors
  1. How sensitive are the checklist results to the number of sampled frames? Would using 20 frames or full-video human evaluation on a subset change the model rankings?

  2. Could the authors provide a few representative failure cases of the GPT-5.5 evaluator, or report agreement statistics beyond correlation?

  3. How robust are the cross-interface conclusions to different prompt formulations or temporal segmentation strategies for language-driven models?

4 comments
AuthorsOpenReview ↗

Response to Reviewer bDho (Part 1)

We sincerely thank you for recognizing the importance of evaluating world reactivity and the breadth of our experimental validation. Your constructive suggestions help us strengthen the full-video human evaluation and test sensitivity to temporal segmentation and prompt formulations.

W1 / Q1: Human annotators and the VLM evaluator observe the same ten sampled frames, potentially missing short interaction events. How sensitive are the results and model rankings to additional frames or full-video human evaluation?

We thank you and Reviewers TP7N and Ce4T for highlighting the potential blind spots of shared frame sampling. We use uniform sampling, as also adopted in benchmarks such as WorldArena [1] and Video-MME [2], with a 10-frame budget to balance temporal coverage and evaluation cost. To address your concern, annotators reassess the full, un-subsampled videos, and we compare VLM scores using 10, 20, and 40 uniformly sampled frames against these human assessments. The overall model ranking remains consistent with the original evaluation under the full-video human assessment.

Correlation with full-video human assessments10 frames20 frames40 frames
Spearman ρ0.83190.81690.8122
Pearson r0.82920.81080.8086

The 10-frame scores correlate with full-video human judgments, and denser sampling does not increase correlation on this set. These results support the use of 10 frames for efficient evaluation. We update the human-alignment results in Section 5.4 (Table 5) and report the frame-budget analysis in Appendix F.5 (Tables 12–13). The pairwise Spearman correlation matrix below reports correlations between the 10-frame scores and those using 20 or 40 frames, suggesting broadly consistent results across frame budgets.

Human10 frames20 frames40 frames
Human1.00000.83190.81690.8122
10 frames0.83191.00000.92750.8938
20 frames0.81690.92751.00000.9117
40 frames0.81220.89380.91171.0000

W2: Cross-interface comparison is not perfectly controlled. Camera, action, and language driven models receive different forms of control, and language models do not have explicit action durations. This makes some direct comparisons harder to interpret, especially when the paper draws conclusions across model paradigms.

We thank you for highlighting this important distinction. We agree that differences in control interfaces should be considered when interpreting cross-paradigm comparisons. Accordingly, our reactivity tasks (Terrain Interaction, Object Interaction, Social Interaction, and Physical Reaction) all use a single atomic control to reduce cross-interface confounding associated with multi-action execution.

For Camera and Subject Control, language-driven prompts specify motion directions and their order without prescribing per-action durations. The evaluator allows action boundaries to follow the generated motion, making the timing requirements less restrictive than those for camera- and action-driven models. Nevertheless, language-driven models still obtain low scores on both tasks. These results suggest difficulties in executing the requested motions and their order, even without fixed per-action timing requirements.

For prompt formulation and evaluation-time temporal segmentation, which are specific to language-driven models in our protocol, we provide robustness analyses in our response to Q3.

AuthorsOpenReview ↗

Response to Reviewer bDho (Part 2)

Q2: Could the authors provide representative failure cases of the GPT-5.5 evaluator, or agreement statistics beyond correlation?

Thank you for suggesting both quantitative agreement statistics and an analysis of evaluator failure cases. We report item-level agreement with human majority-vote labels, alongside correlation with full-video human assessments:

Agreement with full-video human assessments10 frames20 frames40 frames
Spearman ρ0.83190.81690.8122
Pearson r0.82920.81080.8086
Item-level agreement87.29%84.84%85.31%

We report the 10-frame item-level agreement in Section 5.4 and the results for all three frame budgets in Appendix F.5.

Regarding failure cases of the VLM evaluator, our evaluation prompt explicitly states: “use 1 only if the video clearly satisfies the item” and “use 0 if the item is contradicted, missing, ambiguous, partially satisfied, off-screen, or impossible to verify.” Consequently, when visual degradation makes a reaction difficult to judge, the evaluator tends to return False. We provide two examples in the supplementary material. In failure_case_1.mp4, nearby people make room for the protagonist, but occlusion and distorted figures cause the evaluator to mark the crowd-shifting item False. In failure_case_2.mp4, the evaluator recognizes the sign’s displacement after the cart’s impact, but warping and ghosting make it unclear whether its base and legs react first, leading to False for that specific item.

This evaluation policy is conservative when visual evidence is unclear: it avoids assigning high scores when occlusion or ambiguity prevents verification of the response. It also reflects our expectation that world models generate videos in which both the reactions and their causal relationships remain visually coherent and assessable.

AuthorsOpenReview ↗

Response to Reviewer bDho (Part 3)

Q3: How robust are the cross-interface conclusions to different prompt formulations or temporal segmentation strategies for language-driven models?

Thank you for suggesting these robustness checks. We examine prompt formulation and temporal segmentation separately.

(1) Prompt formulation.

We appreciate your suggestion to examine whether more explicit prompts improve control execution. We test three templates on 30 sampled cases, keeping the initial images and requested action sequences fixed. For a three-action Camera Control case, the templates are:

  • Default: “The camera [action 1], then [action 2], then [action 3]. [Scene description]”
  • Explicit order: “First, the camera [action 1]. Next, the camera [action 2]. Finally, the camera [action 3]. [Scene description]”
  • Explicit timing: “During the first third of the video, the camera [action 1]. During the middle third of the video, the camera [action 2]. During the final third of the video, the camera [action 3]. [Scene description]”

We apply the three prompt templates only to language-driven models. Camera- and action-driven scores are obtained under their default settings on the same cases and are included for comparison. Neither explicit ordering nor explicit timing consistently improves performance across models and tasks. In this experiment, the two language-driven models score below the camera- and action-driven references under all three templates.

Camera Control

ModelInterfaceDefaultExplicit orderExplicit timing
NeoVerseCamera95.50——
InSpatio-WorldCamera85.86——
WorldPlayAction94.99——
LingBot-World-V2Action63.06——
Veo 3.1Language42.3636.8628.29
Vidu Q3Language53.2230.3940.35

Subject Control

ModelInterfaceDefaultExplicit orderExplicit timing
WorldPlayAction55.31——
LingBot-World-V2Action72.22——
Veo 3.1Language20.2913.4028.11
Vidu Q3Language36.1143.0236.56

(2) Temporal segmentation.

Thank you for highlighting the potential influence of automatic segmentation. For Camera and Subject Control, which involve multi-action sequences, we ask annotators to segment the language-driven models’ outputs by watching the full videos. We compare scores obtained using these human annotations with those using automatic segmentation. The scores vary modestly under human segmentation, while our main conclusions remain supported. We include this comparison in Table 7 and discuss the results in Section 5.4 of the revised manuscript.

ModelCamera: AutomaticCamera: HumanSubject: AutomaticSubject: Human
Kling 2.550.1854.7728.4030.62
Veo 3.140.8342.7937.2838.81
Hailuo 2.363.2965.1136.4938.25
Wan 2.657.7258.7229.0229.14
Seedance 1.549.1848.9932.5133.80
Vidu Q342.7543.0527.6731.56
HappyHorse 1.058.2958.8333.1132.25

We thank you again for your insightful comments and hope that these clarifications and additional experiments address your concerns.


[1] Shang, Yu, et al. “WorldArena: A Unified Benchmark for Evaluating Perception and Functional Utility of Embodied World Models.” 2026.

[2] Fu, Chaoyou, et al. “Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis.” CVPR, 2025.

Thank for your rebuttal. All of my concerns have been addressed. I will keep my score.

Reviewer_Ce4TOpenReview ↗
Rating
3
Confidence
4(+1)
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

WorldExam examines the distinction between executing a control and generating the plausible consequences implied by a scene. It contains 1,474 cases across eight tasks, organized into visual quality, control adherence, spatial consistency, and world reactivity. Shared control intents are adapted to camera-, action-, and language-driven interfaces. The static track tests camera control and scene revisiting; the dynamic track covers subject motion, terrain adaptation, object and social interactions, physical reactions, and goal completion. Evaluation combines reconstructed geometry with item-level VLM checklists, while reaction-task inputs leave expected consequences unstated. The authors evaluate 23 models and validate the protocol through human agreement and reconstruction-backend comparisons. Their main finding is that strong control does not necessarily imply appropriate reactions, and similar visual quality can conceal substantial behavioral differences.

Critical strengths
  1. Distinguishing control adherence from environmental reactions is useful. The cross-model results show that these capabilities do not always improve together, providing a more specific picture of current world models than a visual-quality score alone. This is my main reason for leaning toward weak acceptance.

  2. The task and interface coverage is broad, and the protocol addresses several practical confounds. Scene Revisit separates failure to return from changes after returning. Displacement calibration, human evaluation, and reconstruction-backend comparisons also make the results easier to interpret.

Critical weaknesses
  1. Some reaction scores mix responses with visibility and coherence. Both examples in Figure 12 pass the core bystander-response item but score 7/7 and 4/7 overall. This is reasonable for end-to-end quality, but attributing the score specifically to reactivity requires separating response items from the others.

  2. Humans and the automated evaluator use the same ten frames (p. 9), potentially missing the same brief contact, interpenetration, or reactions. Their high agreement therefore does not establish temporal correctness throughout the full video. A small blinded full-video study would directly address this gap.

  3. The terrain metric mainly checks height-change trends (Appendix D.3). It is unclear whether a subject hovering along the correct profile could still score well. Changing reconstruction backends is useful, but examples with known ground truth are still needed to check sensitivity to ground-contact errors.

Questions for authors
  1. Do Table 3's main conclusions hold when core response items are reported separately?
  2. Does a small blinded full-video evaluation preserve the model ranking?
  3. How does the terrain metric score ground-truth trajectories modified to hover or penetrate the ground?
3 comments
AuthorsOpenReview ↗

Response to Reviewer Ce4T (Part 1)

We sincerely thank you for your positive assessment and for highlighting the value of distinguishing control adherence from scene-conditioned reactivity, as well as our confound-mitigation designs in Scene Revisit and displacement calibration. Your thoughtful suggestions help us clarify the checklist scores and strengthen evaluator validation.

W1 / Q1: Some reaction scores mix responses with visibility and coherence. Both examples in Figure 12 pass the core bystander-response item but score 7/7 and 4/7 overall. Do Table 3’s main conclusions hold when core response items are reported separately?

Thank you for highlighting this distinction through Figure 12. Both examples satisfy the bystander-response item. Following your suggestion, also raised by Reviewer TP7N, we separate checklist items into Core, assessing target responses, physical consequences, or goal completion, and Supporting, assessing action execution, visibility, identity preservation, and general coherence. The table below reports the original Average scores alongside Core and Supporting scores across all four checklist tasks. The same breakdown is reported in Table 4 and discussed in Section 5.3 of the revised manuscript, with the additions highlighted in red.

ModelObject InteractionSocial InteractionPhysical ReactionGoal Completion
AverageCoreSupportingAverageCoreSupportingAverageCoreSupportingAverageCoreSupporting
Action-driven
WorldPlay33.7521.1743.5751.4034.2657.3626.9121.0744.94———
LingBot-World25.9414.6033.5360.3738.9666.9333.4328.8348.36———
SANA-WM40.1932.4246.2348.7129.8354.9436.4131.8351.18———
LingBot-World-V231.2425.2234.9045.7124.4252.4232.8329.6842.54———
ABot-World36.8829.4542.2048.0827.2955.0139.1735.5050.56———
Language-driven
Kling 2.527.7016.0336.7066.8042.0075.3331.9926.0750.7248.2546.8259.66
Veo 3.175.9664.9784.8785.1078.4886.8461.7659.8266.1685.3084.2492.86
Hailuo 2.367.0162.6070.7272.4544.2580.8063.8459.1577.2078.8677.7990.65
Wan 2.6 I2V44.5634.0352.0566.4032.1377.3548.1542.9362.8876.3975.0890.99
Seedance 1.537.9126.4245.6772.0949.4079.7347.6043.9756.8576.2474.8291.50
Vidu Q371.5967.1874.5981.9164.4687.8661.2358.7066.0578.2677.4189.88
HappyHorse 1.065.7058.0270.9776.1752.9683.5947.0140.4666.0085.3384.4791.41

As shown in the table, core scores are generally lower, providing a clearer assessment of core reactivity. Nevertheless, Core and Average scores exhibit similar ranking patterns (Spearman ρ = 0.972–1.000), supporting our central finding that strong control adherence does not guarantee strong scene-conditioned reactivity.

AuthorsOpenReview ↗

Response to Reviewer Ce4T (Part 2)

W2 / Q2: Humans and the automated evaluator use the same ten frames, potentially missing the same brief events. Does a small blinded full-video evaluation preserve the model ranking?

We thank you and Reviewers TP7N and bDho for highlighting the potential blind spots of shared frame sampling. We use uniform sampling, as also adopted in benchmarks such as WorldArena [1] and Video-MME [2], with a 10-frame budget to balance temporal coverage and evaluation cost. To address your concern, annotators reassess the full, un-subsampled videos, and we compare VLM scores using 10, 20, and 40 uniformly sampled frames against these human assessments. The overall model ranking remains consistent with the original evaluation under the full-video human assessment.

Correlation with full-video human assessments10 frames20 frames40 frames
Spearman ρ0.83190.81690.8122
Pearson r0.82920.81080.8086

The 10-frame scores correlate with full-video human judgments, and denser sampling does not increase correlation on this set. These results support the use of 10 frames for efficient evaluation. We update the human-alignment results in Section 5.4 (Table 5) and report the frame-budget analysis in Appendix F.5 (Tables 12–13). The pairwise Spearman correlation matrix below reports correlations between the 10-frame scores and those using 20 or 40 frames, suggesting broadly consistent results across frame budgets.

Human10 frames20 frames40 frames
Human1.00000.83190.81690.8122
10 frames0.83191.00000.92750.8938
20 frames0.81690.92751.00000.9117
40 frames0.81220.89380.91171.0000

W3 / Q3: Could a subject hovering along the correct terrain profile still score well? How does the terrain metric score ground-truth trajectories modified to hover or penetrate the ground?

We thank you and Reviewer TP7N for the helpful suggestion to examine subject–terrain contact using controlled trajectories. We consider this issue during benchmark construction and curate the initial images to place the subject in contact with the supporting terrain. Thus, the specific scenario of “a subject hovering along the correct profile” from an initially floating position is excluded. Starting from this state, our task examines whether a horizontal movement instruction induces appropriate vertical adaptation to uneven terrain.

Contact checking is a valuable consideration. However, valid behaviors such as crossing an obstacle may involve jumping before reaching it and temporarily losing contact. We therefore use height trends to evaluate whether the subject responds appropriately to changes in terrain elevation, without requiring contact with the terrain at every frame.

To illustrate the metric’s behavior, we construct Three.js scenes spanning five terrain types and three subject types with programmatically specified trajectories. We render 30 good–bad video pairs from these scenes, pairing terrain-following trajectories with deviations involving hovering, penetration, or only partial terrain adaptation. For these paired examples, the scores are consistent with the intended trajectory differences. Representative cases and their scores appear in Appendix F.4.

We thank you again for your insightful comments and hope that these clarifications and additional experiments address your concerns.


[1] Shang, Yu, et al. “WorldArena: A Unified Benchmark for Evaluating Perception and Functional Utility of Embodied World Models.” 2026.

[2] Fu, Chaoyou, et al. “Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis.” CVPR, 2025.

Thanks for your detailed reply and your updated manuscript. My concerns have been addressed. Overall, the research topic is interesting, and the paper’s contributions are substantial, so I will maintain my score and increase my confidence in the assessment.

Reviewer_TP7NOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

WorldExam evaluates video world models on eight tasks covering control, spatial consistency, and responses to the surrounding scene. It adapts inputs to different control interfaces and combines geometric measurements with checklist-based evaluation.

Critical strengths
  • Testing consequences that are not explicitly requested is useful. It helps distinguish following a command from generating a plausible response to the scene.
  • The benchmark covers 1,474 cases and 23 models. Human-agreement and reconstruction-backend checks provide useful evidence about evaluation reliability.
Critical weaknesses
  • Some scores are indirect measures of the claimed capability. Terrain evaluation captures height trends without establishing correct contact, while checklist averaging can reward a video that misses the central physical event. Could the authors audit these failure cases and test an aggregation that requires the essential consequence?
  • Human–VLM agreement uses the same ten sampled frames, so both can miss brief events. Please include a full-video human audit and direct validation of the geometric metrics. This is particularly important for establishing what WorldExam measures beyond WorldScore and WorldModelBench.
Questions for authors

See weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer TP7N (Part 1)

We sincerely thank you for recognizing the value of evaluating unstated consequences and the breadth of WorldExam. Your constructive suggestions help us sharpen the evaluation of core reactivity and strengthen its validation.

W1: Some scores are indirect measures of the claimed capability. Terrain evaluation captures height trends without establishing correct contact, while checklist averaging can reward a video that misses the central physical event. Could the authors audit these failure cases and test an aggregation that requires the essential consequence?

(1) Terrain interaction: rationale and controlled validation.

We thank you for your careful reading and the helpful suggestion regarding subject–terrain contact. We consider this issue during benchmark construction and curate the initial images to place the subject in contact with the supporting terrain. Starting from this state, our task examines whether a horizontal movement instruction induces appropriate vertical adaptation to uneven terrain.

Contact checking is a valuable consideration. However, valid behaviors such as crossing an obstacle may involve jumping before reaching it and temporarily losing contact. We therefore use height trends to evaluate whether the subject responds appropriately to changes in terrain elevation, without requiring contact with the terrain at every frame.

To illustrate the metric’s behavior, we construct Three.js scenes spanning five terrain types and three subject types with programmatically specified trajectories. We render 30 good–bad video pairs from these scenes, pairing terrain-following trajectories with deviations involving hovering, penetration, or only partial terrain adaptation. For these paired examples, the scores are consistent with the intended trajectory differences. Representative cases and their scores appear in Appendix F.4.

(2) Checklist evaluation: core and supporting breakdown.

Thank you for highlighting how supporting criteria can obscure a missing core response. We separate checklist items into Core, assessing target responses, physical consequences, or goal completion, and Supporting, assessing action execution, visibility, identity preservation, and general coherence. The Physical Reaction bad case in Appendix B.7 (Fig. 13b) shows the basket returning to a supported position instead of falling. It fails all six Core items and passes the Supporting item requiring the woman to remain stationary, yielding an Average score of 14.29 (1/7) and a Core score of 0. The zero Core score makes the missing physical response explicit in this example.

The table below reports the original Average scores alongside Core and Supporting scores across all four checklist tasks. The same breakdown is reported in Table 4 and discussed in Section 5.3 of the revised manuscript.

ModelObject InteractionSocial InteractionPhysical ReactionGoal Completion
AverageCoreSupportingAverageCoreSupportingAverageCoreSupportingAverageCoreSupporting
Action-driven
WorldPlay33.7521.1743.5751.4034.2657.3626.9121.0744.94———
LingBot-World25.9414.6033.5360.3738.9666.9333.4328.8348.36———
SANA-WM40.1932.4246.2348.7129.8354.9436.4131.8351.18———
LingBot-World-V231.2425.2234.9045.7124.4252.4232.8329.6842.54———
ABot-World36.8829.4542.2048.0827.2955.0139.1735.5050.56———
Language-driven
Kling 2.527.7016.0336.7066.8042.0075.3331.9926.0750.7248.2546.8259.66
Veo 3.175.9664.9784.8785.1078.4886.8461.7659.8266.1685.3084.2492.86
Hailuo 2.367.0162.6070.7272.4544.2580.8063.8459.1577.2078.8677.7990.65
Wan 2.6 I2V44.5634.0352.0566.4032.1377.3548.1542.9362.8876.3975.0890.99
Seedance 1.537.9126.4245.6772.0949.4079.7347.6043.9756.8576.2474.8291.50
Vidu Q371.5967.1874.5981.9164.4687.8661.2358.7066.0578.2677.4189.88
HappyHorse 1.065.7058.0270.9776.1752.9683.5947.0140.4666.0085.3384.4791.41

As shown in the table, core scores are generally lower, providing a clearer assessment of core reactivity. Nevertheless, Core and Average scores exhibit similar ranking patterns (Spearman ρ = 0.972–1.000), supporting our central finding that strong control adherence does not guarantee strong scene-conditioned reactivity.

AuthorsOpenReview ↗

Response to Reviewer TP7N (Part 2)

W2: Human–VLM agreement uses the same ten sampled frames, so both can miss brief events. Please include a full-video human audit and direct validation of the geometric metrics. This is particularly important for establishing what WorldExam measures beyond WorldScore and WorldModelBench.

(1) Full-video human audit across frame budgets.

We thank you and Reviewers Ce4T and bDho for highlighting the potential blind spots of frame sampling. We use uniform sampling, as also adopted in benchmarks such as WorldArena [1] and Video-MME [2], with a 10-frame budget to balance temporal coverage and evaluation cost. To directly address your concern, annotators reassess the full, un-subsampled videos, and we compare VLM scores using 10, 20, and 40 uniformly sampled frames against these human assessments.

Correlation with full-video human assessments10 frames20 frames40 frames
Spearman ρ0.83190.81690.8122
Pearson r0.82920.81080.8086

The 10-frame scores correlate with full-video human judgments, and denser sampling does not increase correlation on this set. These results support the use of 10 frames for efficient evaluation. We update the human-alignment results in Section 5.4 (Table 5) and report the frame-budget analysis in Appendix F.5 (Tables 12–13). The pairwise Spearman correlation matrix below reports correlations between the 10-frame scores and those using 20 or 40 frames, suggesting broadly consistent results across frame budgets.

Human10 frames20 frames40 frames
Human1.00000.83190.81690.8122
10 frames0.83191.00000.92750.8938
20 frames0.81690.92751.00000.9117
40 frames0.81220.89380.91171.0000

(2) Direct validation of geometric metrics.

We appreciate your suggestion to validate the geometric evaluation against known trajectories. As detailed in W1(1), we construct controlled Three.js scenes with programmatically specified terrain-adaptive, hovering, and penetrating trajectories. Representative paired examples receive scores of 100 for terrain adaptation and 0 or 50 for absent or incomplete adaptation. These examples and their scores are included in Appendix F.4. Together with the Core-only analysis in W1, these additional experiments provide further evidence regarding evaluator reliability through full-video human assessments and controlled terrain trajectories.

We thank you again for the insightful comments and hope that these clarifications and additional experiments address your concerns.


[1] Shang, Yu, et al. "Worldarena: A unified benchmark for evaluating perception and functional utility of embodied world models." arXiv preprint arXiv:2602.08971 (2026).

[2] Fu, Chaoyou, et al. "Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Brief summary and sincere thanks to all reviewers

Thank you all for taking the time to read our paper and share your thoughtful suggestions. We are delighted that all three reviewers have given positive ratings and recognized the value of evaluating scene-implied consequences beyond explicit instruction following. Your feedback has helped us sharpen the distinction between control adherence and scene-conditioned reactivity and strengthen the validation of our evaluation protocol.

  • Reviewer TP7N highlighted the value of evaluating consequences that are not explicitly requested and the breadth of our benchmark. In response to concerns about what the metrics capture, we added a Core/Supporting checklist breakdown and controlled terrain examples with known trajectories. Core and original Average scores exhibit similar ranking patterns, while the breakdown makes failures of essential reactions more explicit.

  • Reviewer Ce4T recognized the importance of separating control adherence from environmental reactions and appreciated our efforts to address evaluation confounds. We added comparisons against full-video human assessments across 10-, 20-, and 40-frame evaluation budgets, alongside the checklist breakdown and controlled terrain validation. We especially appreciate your confirmation that your concerns have been addressed, your continued positive rating, and your increased confidence in the assessment.

  • Reviewer bDho emphasized the importance of world reactivity and the broad experimental coverage. We supplemented the frame-budget analysis with item-level agreement statistics and evaluator failure cases, clarified differences between control interfaces, and examined sensitivity to prompt formulation and human-annotated temporal segmentation. We greatly appreciate your follow-up confirming that all your concerns have been addressed and that you will maintain your positive rating.

The additional analyses and clarifications are documented in our revised manuscript, detailed responses, and supplementary material. Thank you again for your time and care throughout the review process. Your feedback has helped us make WorldExam clearer and strengthen its evaluation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.