- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
WorldBench contains 112 multi-shot tasks across seven indoor scene types, covering state persistence, participant entry, participant exit, and position swaps. It uses a valid generated opening as a reference to determine the expected participants and their locations at the end, then checks the final shot from a different viewpoint. The paper compares nine models and examines the effects of participant count and shot count.
Critical strengths
1.Multi-shot videos need to preserve established states while making the changes required by the narrative. The paper brings these requirements into a single evaluation, providing a useful tool for studying the limitations of current video models. 2.The paper compares models across different tasks and difficulty settings. The breakdown of results and failure examples helps readers understand the conditions under which models tend to struggle.
Critical weaknesses
-
The paper does not specify the video duration for each model or whether videos are generated in one pass or shot by shot. It also uses FFmpeg to identify shot boundaries, but does not explain how videos with no clear cuts or an unexpected number of shots are handled. Clarifying these details would help readers better understand and reproduce the results.
-
Figure 4 shows that success rates decrease with more participants or shots. However, overall success depends on both observation validity and final-state correctness. Although the paper reports these metrics separately in Table 1, a similar breakdown for Figure 4 would help explain what causes the performance drops.
Questions for authors
Could the authors clarify the video duration and generation process for each model? How are videos with no clear cuts or an unexpected number of shots handled? Could errors in FFmpeg shot detection affect the evaluation results? Section 2 discusses the differences from MSVBench and StateBench. Could the authors give a concrete example of what WorldBench captures that these benchmarks might miss?
Response to Reviewer Seev
We sincerely thank the reviewer for the careful examination of our paper and released artifacts, and for highlighting these important concerns.
Clarification of the Released Code
We apologize for mistakenly releasing an outdated development version of our code in the anonymous repository. During iterative development and debugging, we experimented with alternative language models, including Gemini 3.7 Flash and GPT-5.6 Luna, to reduce development costs. Unfortunately, the initially released repository contained artifacts from this intermediate version, leading to the discrepancies identified by the reviewer.
We have now updated the anonymous repository with the version corresponding to the experiments reported in the manuscript. The
benchmark/outputs/directory contains the benchmark dataset used in our experiments. For transparency and traceability, we have retained the previous v3.1 artifacts rather than silently removing them. In addition, we have provided generated video outputs from the official APIs of three evaluated models in theresults/directory to make the experimental setting and representative generation behaviors easier to inspect.We sincerely apologize for the confusion caused by this release error and greatly appreciate the reviewer bringing it to our attention.
Motivation for Endpoint-Based Evaluation
We appreciate the reviewer's interest in our endpoint-based evaluation design and acknowledge its limitations, particularly the possibility that inconsistencies in intermediate shots may go undetected. By elaborating on the motivations behind this design choice, we hope to provide a clearer understanding of the challenges encountered during benchmark development and the considerations that shaped our evaluation protocol.
Our primary motivation is to assess whether multi-shot video generators can maintain previously established world states while selectively incorporating narrative updates, rather than exhaustively detecting visual inconsistencies across shots. This motivation stems from our observations that current models frequently exhibit inconsistencies in subject presence and spatial assignments across shots, particularly when certain subjects are not explicitly mentioned in subsequent prompts. These failures raise the question of whether existing generators can reliably preserve world-state information about subjects that are temporarily unobserved or unmentioned.
However, evaluating world-state consistency across every intermediate shot introduces a fundamental ambiguity: the absence of a subject from a particular view does not necessarily indicate a failure to maintain the underlying world state, as the subject may simply fall outside the camera's field of view. Moreover, attempting to capture all possible inconsistencies involving subjects, objects, and scene geometry would broaden the evaluation into a general visual continuity detection task, deviating from our focus on selective state maintenance.
To address these challenges, we adopt a controlled endpoint-based evaluation protocol. The opening wide shot establishes an observable reference state, while intermediate narrative instructions specify the intended state transitions. The final wide shot then re-observes the scene, allowing us to compare the resulting state against the expected state derived from the opening observation and the prescribed updates. This design enables us to assess whether the requested changes are correctly applied while unaffected state components are preserved. Importantly, we require the final wide shot to adopt a different viewpoint, such as a reverse or overhead view, to discourage trivial frame repetition and evaluate state consistency across viewpoint changes.
We recognize this limitation as a trade-off of our endpoint-based evaluation design. The constructive feedback from all reviewers is sincerely appreciated and will help guide further improvements to our work.