acceptodds
Withdrawn from ICLR 2027

WorldBench: Selective State Maintenance in Multi-Shot Video Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent advances in video generation have enabled increasingly coherent multi-shot video generation with controllable camera viewpoints and recurring subjects. While recent multi-shot video generators have made substantial progress in cross-shot visual consistency, they still frequently exhibit world-state collapse, leading to inconsistencies in subject count and spatial assignments across shots. Motivated by this gap, we introduce WorldBench, a controlled benchmark for evaluating world-state maintenance in multi-shot video generation. Each episode establishes an initial world state with a wide shot and later re-observes the resulting world from a different viewpoint after controlled narrative changes. Experiments on representative open- and closed-weight models show that even Seedance 2.5 achieves only 56.2% end-to-end success, confirming that world-state maintenance remains a substantial challenge and highlighting the significance of WorldBench for systematically evaluating this capability. Our code is available at https://anonymous.4open.science/r/WorldBench-1.

Primary area:
datasets and benchmarks
Keywords:
video generation, world state

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_mfKEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

WorldBench contains 112 multi-shot tasks across seven indoor scene types, covering state persistence, participant entry, participant exit, and position swaps. It uses a valid generated opening as a reference to determine the expected participants and their locations at the end, then checks the final shot from a different viewpoint. The paper compares nine models and examines the effects of participant count and shot count.

Critical strengths

1.Multi-shot videos need to preserve established states while making the changes required by the narrative. The paper brings these requirements into a single evaluation, providing a useful tool for studying the limitations of current video models. 2.The paper compares models across different tasks and difficulty settings. The breakdown of results and failure examples helps readers understand the conditions under which models tend to struggle.

Critical weaknesses
  1. The paper does not specify the video duration for each model or whether videos are generated in one pass or shot by shot. It also uses FFmpeg to identify shot boundaries, but does not explain how videos with no clear cuts or an unexpected number of shots are handled. Clarifying these details would help readers better understand and reproduce the results.

  2. Figure 4 shows that success rates decrease with more participants or shots. However, overall success depends on both observation validity and final-state correctness. Although the paper reports these metrics separately in Table 1, a similar breakdown for Figure 4 would help explain what causes the performance drops.

Questions for authors

Could the authors clarify the video duration and generation process for each model? How are videos with no clear cuts or an unexpected number of shots handled? Could errors in FFmpeg shot detection affect the evaluation results? Section 2 discusses the differences from MSVBench and StateBench. Could the authors give a concrete example of what WorldBench captures that these benchmarks might miss?

Reviewer_SeevOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces WorldBench, a benchmark for selective world-state maintenance in multi-shot video generation. Its 112 episodes cover seven indoor scene types and four transitions: Static, Entry, Exit, and Position Swap. An opening wide shot establishes the generated cast and physical locations, intermediate shots introduce a prescribed update, and a final reverse or overhead view is used to recover the resulting state. GPT-5.6 Sol extracts structured records from sampled opening and final frames, from which Valid, Count, Position, and end-to-end success rate are computed. Experiments evaluate nine open- and closed-weight systems and report a highest success rate of 56.2% for Seedance 2.5.

Critical strengths
  1. The benchmark asks a useful and more specific question than generic cross-shot consistency: whether a requested state change is executed while all unaffected people and locations are preserved. Deriving the target from the generated opening rather than a prescribed reference layout is a sensible design choice because it separates the state realized by the model from the camera view used to observe it.
  2. The transition-wise analysis is potentially diagnostic. Under the proposed evaluator, Persistence is much easier than selective updates, while Position Swap is consistently the hardest transition. The accompanying failure cases make the intended distinction between incorrect counts, failed updates, and changed physical layouts easy to understand.
Critical weaknesses
  1. The reliability of the benchmark’s central evaluator has not been established. Lines 355–356 identify GPT-5.6 Sol as the multimodal evaluator, while Figure 3 uses a single joint call that receives the opening and final evidence together with the requested update. The paper reports no human agreement, repeated-run stability, or criterion-level error analysis for counting, cross-view identity matching, and physical-place correspondence. The released scoring protocol itself notes that final evidence may influence the recovered opening state and that human calibration is required. There is also an unresolved inconsistency: benchmark/README.md states that human review and judge calibration remain pending, whereas lines 248–251 state that prompt generation was followed by manual review and lines 488–491 state that AI-assisted outputs were reviewed and verified. A released Core v3.1 run.json also records Gemini 3.7 Flash for four construction stages and GPT-5.6 Luna for prompt auditing, rather than GPT-5.6 Sol being used throughout as stated in lines 248–251. These discrepancies leave the evaluator’s validation status and the provenance of the released benchmark unclear.
  2. The reported metrics do not cleanly isolate world-state maintenance. Lines 287–290 reject an opening when the intended cast is not established, lines 310–313 reject a final observation when the scene is not sufficiently observable, and Eq. (11) at lines 342–347 counts both cases as SR failures. SR therefore combines opening prompt adherence, final-shot observability, and state correctness. Reporting Valid separately is helpful, but Count and Position are then computed on different model-specific valid subsets at lines 334–341, so cross-model comparisons may be affected by differences in the difficulty of each model’s valid subset. The protocol also evaluates only the opening and two frames from the final shot, as shown in Figure 3 and lines 287–325. Intermediate disappearances, duplications, or incorrect assignments can therefore go undetected if the expected final state is recovered. The benchmark directly measures endpoint state correctness, but provides limited evidence about state consistency between those endpoints. The evaluated state space is also limited to participant presence and coarse physical-place assignments in small indoor groups, so the conclusions should not be generalized to world-state maintenance as a whole. The comparison at lines 426–428 does not isolate the effect of shot count: the released Core v3.1 manifest at benchmark/outputs/v3.1/core_matrix.manifest.json assigns all 56 three-shot episodes to seven source excerpts and all 56 four-shot episodes to seven disjoint source excerpts. Consequently, the reported 11.6-point decrease can reflect story content as well as sequence length. These confounds limit the interpretation of the headline success rates, cross-model rankings, and shot-count analysis.
Questions for authors
  1. Please clarify what evidence supports the validity of WorldBench and its conclusions, given that the released benchmark states that human review and judge calibration remain pending and the paper reports no human agreement, evaluator repeatability, or criterion-level errors for Valid, Count, and Position.
  2. Which frozen dataset, construction records, and scoring protocol were used to produce the reported results? Please clarify how the construction-model records in the released Core v3.1 files relate to the protocol described in the paper, and whether the potential influence of final evidence on opening-state recovery was assessed.
  3. How much of each model’s SR failure comes from invalid openings, unobservable final shots, count errors, and position errors? Do the relative conclusions persist when state correctness is evaluated on a comparable set of episodes?
  4. Does the reported effect of shot count remain when three-shot and four-shot episodes are matched by story source and the other controlled factors?
1 comment
AuthorsOpenReview ↗

Response to Reviewer Seev

We sincerely thank the reviewer for the careful examination of our paper and released artifacts, and for highlighting these important concerns.

Clarification of the Released Code

We apologize for mistakenly releasing an outdated development version of our code in the anonymous repository. During iterative development and debugging, we experimented with alternative language models, including Gemini 3.7 Flash and GPT-5.6 Luna, to reduce development costs. Unfortunately, the initially released repository contained artifacts from this intermediate version, leading to the discrepancies identified by the reviewer.

We have now updated the anonymous repository with the version corresponding to the experiments reported in the manuscript. The benchmark/outputs/ directory contains the benchmark dataset used in our experiments. For transparency and traceability, we have retained the previous v3.1 artifacts rather than silently removing them. In addition, we have provided generated video outputs from the official APIs of three evaluated models in the results/ directory to make the experimental setting and representative generation behaviors easier to inspect.

We sincerely apologize for the confusion caused by this release error and greatly appreciate the reviewer bringing it to our attention.

Motivation for Endpoint-Based Evaluation

We appreciate the reviewer's interest in our endpoint-based evaluation design and acknowledge its limitations, particularly the possibility that inconsistencies in intermediate shots may go undetected. By elaborating on the motivations behind this design choice, we hope to provide a clearer understanding of the challenges encountered during benchmark development and the considerations that shaped our evaluation protocol.

Our primary motivation is to assess whether multi-shot video generators can maintain previously established world states while selectively incorporating narrative updates, rather than exhaustively detecting visual inconsistencies across shots. This motivation stems from our observations that current models frequently exhibit inconsistencies in subject presence and spatial assignments across shots, particularly when certain subjects are not explicitly mentioned in subsequent prompts. These failures raise the question of whether existing generators can reliably preserve world-state information about subjects that are temporarily unobserved or unmentioned.

However, evaluating world-state consistency across every intermediate shot introduces a fundamental ambiguity: the absence of a subject from a particular view does not necessarily indicate a failure to maintain the underlying world state, as the subject may simply fall outside the camera's field of view. Moreover, attempting to capture all possible inconsistencies involving subjects, objects, and scene geometry would broaden the evaluation into a general visual continuity detection task, deviating from our focus on selective state maintenance.

To address these challenges, we adopt a controlled endpoint-based evaluation protocol. The opening wide shot establishes an observable reference state, while intermediate narrative instructions specify the intended state transitions. The final wide shot then re-observes the scene, allowing us to compare the resulting state against the expected state derived from the opening observation and the prescribed updates. This design enables us to assess whether the requested changes are correctly applied while unaffected state components are preserved. Importantly, we require the final wide shot to adopt a different viewpoint, such as a reverse or overhead view, to discourage trivial frame repetition and evaluate state consistency across viewpoint changes.

We recognize this limitation as a trade-off of our endpoint-based evaluation design. The constructive feedback from all reviewers is sincerely appreciated and will help guide further improvements to our work.

Reviewer_YkViOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces WorldBench, a benchmark for evaluating selective world-state maintenance in multi-shot video generation. The benchmark constructs 112 story-grounded episodes across seven indoor scene types and four state transitions: static persistence, subject entry, subject exit, and position swap. Each episode contains an opening wide shot that establishes a reference world state, intermediate narrative shots, and a final shot from a different viewpoint that re-observes the scene. The expected final state is derived by applying the requested narrative updates to the opening state, while preserving unaffected subjects and spatial assignments. The evaluation separates observation validity from state correctness. It reports the fraction of episodes with sufficiently observable opening and final shots, count correctness, position correctness, and an end-to-end success rate. Experiments on several open- and closed-weight video generators show substantial failures in subject preservation and spatial assignment. Seedance 2.5 achieves the best overall success rate at 56.2%, while position swaps are consistently the most difficult transition. The paper also reports degradation with increasing subject count and shot count and analyzes representative failure modes.

Critical strengths
  1. A useful and clearly defined evaluation target. WorldBench isolates selective state maintenance: models must preserve unaffected entities and locations while applying an explicit update. This is a meaningful capability beyond simple visual similarity or generic cross-shot consistency.

  2. The evaluation separates observation failure from state failure. Reporting Valid, Count, Position, and end-to-end SR is a sensible design choice. It avoids interpreting an unobservable final crop as evidence that a model incorrectly removed a subject, while still exposing the practical cost of producing evaluable, coherent shots.

Critical weaknesses
  1. The benchmark and its conclusions rely heavily on a single LLM evaluator without sufficient validation. GPT-5.6 Sol is used to identify subjects, infer physical locations, determine viewpoint validity, and compare opening and final states. The paper does not report human agreement, evaluator accuracy, calibration, sensitivity to frame choice, or comparisons with independent annotators. Since the benchmark’s central metrics depend on these judgments, evaluator errors could materially affect the reported rankings and failure rates.

  2. The benchmark is small and narrow, and the results lack statistical uncertainty. There are only 112 episodes across 28 scene-transition configurations, implying very few episodes per configuration. The paper appears to use one generation per model and episode, but does not report repeated samples, confidence intervals, or significance tests. The benchmark also covers only indoor scenes, 3–4 subjects, 3–4 shots, subject presence, and coarse place assignments. Consequently, the strong conclusions about general multi-shot world-state maintenance may be broader than what the benchmark directly supports.

  3. The end-to-end success rate conflates initial scene establishment, camera compliance, and state maintenance. An episode is invalid when the opening cast is wrong or the final view is insufficiently observable. This is a useful practical metric, but it means SR measures a combination of prompt following, shot composition, viewpoint coverage, and state maintenance. The conditional Count and Position metrics partially address this issue, yet the paper’s headline conclusions should distinguish these capabilities more explicitly.

Questions for authors
  1. How many independent generations were produced for each model and episode? If only one sample was used, can you provide confidence intervals or repeated-generation results to quantify stochastic variation?

  2. What is the accuracy and inter-rater agreement of the GPT-5.6-based evaluator? Were any episodes independently annotated by human reviewers, and how often did the evaluator disagree with them?

  3. How were prompts and generation parameters adapted for each baseline, especially models that do not natively support the same multi-shot or viewpoint-control interface? Please report all prompt templates, seeds, duration settings, resolutions, sampling parameters, and API/model versions.

  4. Since the opening shot is itself generated by the evaluated model, how much of the overall SR difference is explained by invalid initial casts or unobservable final views rather than failures to maintain a valid state? A decomposition with fixed or human-provided opening references would clarify this point.

  5. Can the authors evaluate the benchmark on broader state types, such as object states, continuous motion, occlusion followed by reappearance, or more than four subjects, to determine whether the observed trends extend beyond the current controlled setting?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.