acceptodds
Withdrawn from ICLR 2027

Do Video Generators Track the World Across Segments? A Benchmark and Method for World-State Reasoning in Video Continuation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video generators build long videos by composing shorter parts, either by generating segments one after another or by autoregressively extending chunks. Each new part usually depends on memories of historical observations, such as recent frames, selected key frames, memory banks, or cached features. These memories preserve visible evidence from the past, but current generators do not reliably turn such evidence into a world-state interface: what holds in the video world after previous actions and how it should change under the next prompt. A past frame remains valid history, but it may not describe the state needed by the next segment; some states must instead be inferred from occluded or implicit changes rather than copied from a directly observed frame. This creates a simple but overlooked question for video continuation: given a previous video, its prompt, and a new prompt, can a model generate a continuation that reflects the state determined by both the historical video and the new prompt? To answer this question, we introduce Statebench, a benchmark that targets this gap by testing continuations over three state categories: past-visible states, occluded-process states, and complex-transition states. We further propose Stateagent, which explicitly maintains an entity-state representation, updates it under the new prompt, grounds the predicted post-action state as a future end frame, and renders the next video. Experiments show that our method improves controlled video continuation by raising the all-case state score (SCS-All) from 45.4 to 69.3, and also benefits story generation at the one-minute scale.

Primary area:
generative models
Keywords:
Video Continuation, Video Generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_dea7OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces StateBench, a 200-case benchmark for world-state reasoning across video segments, covering past-visible states, occluded processes, and complex transitions. Its evaluation separates completion of the requested action from state correctness and penalizes exposure failures in an all-case score. The paper also proposes StateAgent, which parses and updates structured entity states, grounds them into a future frame through image editing, and generates the continuation. Experiments report an improvement in the all-case score from 45.4 to 69.3, with additional evaluation on minute-long storytelling.

Critical strengths
  1. A focused and consequential evaluation target. The benchmark tests whether continuations preserve states established before a segment boundary, addressing an important limitation that appearance consistency alone does not capture.

  2. Exposure-aware evaluation. Separating action completion from state correctness helps distinguish failures to reveal the state from incorrect revealed states. The all-case score prevents high conditional scores from masking exposure failures.

  3. A constructive baseline with substantial reported gains. StateAgent provides inspectable intermediate representations and improves the all-case score from 45.4 to 69.3. Comparisons with direct frame editing and textual state conditioning provide useful evidence for the proposed design.

Critical weaknesses
  1. Insufficient documentation of benchmark construction and reference-state validity. The provided version does not adequately specify the provenance, selection, and validation of historical videos. It is therefore difficult to assess whether the reference states are clearly supported by the actual videos, particularly for occluded or complex transitions.

  2. Insufficient evaluator specification and human calibration. The evaluation model, prompts, and sampling settings are insufficiently documented. Agreement with human judgments is not reported, limiting the reproducibility and reliability of the scores.

  3. Limited evidence for long-horizon benefits and result stability. The Bench consistency gain is small (0.586 to 0.594), and its connection to sustained physical-state consistency remains unclear. Repeated-run results or uncertainty estimates are missing, leaving the robustness of the long-horizon gains uncertain.

Questions for authors

see above

Reviewer_zJQROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the problem of world-state consistency in cross-segment video continuation and proposes a state-aware video generation framework and related benchmark.

Critical strengths
  1. The problem is clearly and precisely defined.
  2. The proposed method is simple yet effective, and demonstrates strong performance on the curated benchmark.
Critical weaknesses
  1. The authors have not included the three statement sections required and encouraged by ICLR after the main text.
  2. The role of the history prompt has not been fully evaluated; could additional experiments be conducted to demonstrate that the historical information is primarily derived from the memory of visual states?
  3. The scale of the experiments is limited, and the video distribution is too narrow, falling short of what is expected for a benchmark.
  4. As a method that involves multiple calls to various VLMs, the analysis of inference efficiency is insufficient.
Questions for authors
  1. Could you provide additional experiments to address the concerns raised above?
  2. Could you provide a more detailed efficiency analysis?
Reviewer_g8SbOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether video generators can preserve and update world states across segment boundaries, introducing StateBench, a benchmark of 200 continuation tasks with fixed historical videos. It proposes StateAgent, which parses an entity-state graph, predicts the target state under a new prompt, grounds that state into a future end frame, and generates the continuation using a first–last-frame video model. Experiments show improved state-consistency scores on StateBench and additional gains in one-minute story generation.

Critical strengths
  1. A useful controlled evaluation of history-dependent continuation. Providing a fixed historical video helps separate failures in establishing an earlier event from failures in using its consequences during continuation. The evaluation also distinguishes successful state exposure from state correctness, preventing models from avoiding errors simply by failing to reveal the relevant content.

  2. A practical and interpretable framework with clear empirical gains. The separation of state parsing, state updating, visual grounding, and video rendering makes intermediate predictions inspectable and does not require retraining the video generator. The improvement in SCS-All from 45.4 to 69.3 is substantial, and the ablations support the benefit of combining state reasoning with visual grounding.

Critical weaknesses
  1. The conceptual and methodological novelty appears limited. World-state reasoning and hidden-state evolution have already been investigated by related methods and benchmarks. For example, STEVO-Bench evaluates state evolution under interrupted observation, WorldReasonBench tests future-state prediction given initial conditions and actions, and LiveWorld explicitly maintains evolving states beyond the current field of view. Thus, the broader motivation of moving beyond visual memory toward state tracking is already well established. The fixed-history, cross-segment protocol is a meaningful distinction, but the paper does not sufficiently establish whether it reveals qualitatively different failure modes from existing evaluations. Methodologically, StateAgent primarily combines existing VLM reasoning, image editing, and keyframe-to-video generation capabilities. The current ablations do not isolate whether the explicit entity-state graph and staged update provide benefits over simpler alternatives using the same models, such as a free-form history summary or direct VLM-generated editing instructions.

  2. The generality and scalability of the proposed solution remain insufficiently established. The framework provides an explicit target-state constraint, but its ability to handle complex scenes and continuous dynamics is unclear. In dense scenes where many entities are individually relevant—for example, people exchanging objects in a crowd or visually similar vehicles interacting under occlusion—a sparse graph may discard necessary identity and relational information, while retaining each entity may substantially increase state-management and appearance-binding demands. Although static buildings or trees need not be represented individually, the paper does not establish how entity selection and representation granularity adapt when many such elements become relevant. Ongoing actions present a related limitation: they can be described textually, but a predicted end frame does not uniquely specify velocity, motion phase, contact dynamics, or intermediate event order. For example, nearly identical boundary images may show a ball moving upward or downward, requiring different continuations. Since the final video renderer receives the boundary frame, predicted end frame, and continuation prompt rather than the complete historical dynamics, the method does not explicitly ensure that these distinctions survive the intermediate representation. Correct endpoints therefore do not establish a physically consistent trajectory. Reference selection and computational cost also require clarification. The paper associates entities with historical frames but does not sufficiently specify selection, deduplication, reference budgets, or handling of outdated appearances. If many relevant entities require evidence from different historical moments, the selected references could approach the full sampled history, shifting the context burden to the upstream modules. The additional VLM processing and image-generation stage introduce inference overhead that is not quantified. The one-minute story experiments provide useful evidence beyond the controlled benchmark, but do not resolve these scalability and continuous-dynamics concerns.

Questions for authors
  1. With the VLM, image editor, video renderer, and inference budget held fixed, how does the explicit entity-state graph compare with a free-form history summary or direct generation of editing instructions from the historical video? Can these controls establish the independent contribution of the structured representation and state-update stage?

  2. How are entities and historical appearance references selected, retained, and updated as history grows? Could the authors report performance, reference counts, and end-to-end latency or cost as the number of relevant entities increases, and evaluate ongoing-motion scenarios using trajectory and cross-boundary motion consistency rather than target-state correctness alone?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.