acceptodds
Withdrawn from ICLR 2027

WorldProbe: When Video World Models Fall Apart under Long-Horizon Spatial Exploration

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video world models generate future visual observations from camera or action inputs, with the goal of synthesizing persistent 3D worlds that can be explored across time and space. Existing benchmarks typically evaluate short interactions or rely on camera trajectories fixed before generation. The performance of video world models under long-horizon exploration with extensive camera motion remains under-explored. We introduce WorldProbe, a diagnostic benchmark that uses a goal-conditioned multimodal agent to adapt camera actions online during generation. WorldProbe comprises 164 test cases across four probing families. Across 10 state-of-the-art models, we generate more than 1,600 rollouts totaling approximately 800 minutes, with horizons up to 180 seconds. Our evaluation across four dimensions reveals both the degradation of local measures and the emergence of long-range inconsistencies. We find that recent models sustain visual and content quality for longer, while long-range memory and evolution remain weak. These results expose a gap between locally stable generation and persistent world consistency that is largely missed by existing benchmarks.

Primary area:
datasets and benchmarks
Keywords:
Video World Models, Controllable Video Generation, Generative Model Evaluation, Spatiotemporal Consistency, Long-Horizon Generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_BAWTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces WorldProbe, a benchmark that asks whether video world models remain coherent when exploration responds to what they generate. A goal-conditioned agent chooses camera actions online, assisted by target tracking and local geometry estimates. The benchmark contains 164 cases covering target approach, open-ended traversal, cross-view inspection, and successive region exploration, with task budgets up to 180 seconds. Experiments on ten models examine local quality, spatial memory, and evolution fidelity. The results document failures in scene persistence and coherent evolution during extended exploration, including cases where individual frames remain visually plausible.

Critical strengths

The exploration tasks provide a useful way to test whether a generated world supports sustained, goal-directed navigation. Adjusting actions to newly revealed geometry can help an evaluator continue toward a meaningful observation point when a predefined sequence would no longer reach it. The four task families give this idea a concrete and reusable evaluation setting.

The study across ten models provides useful examples of how local visual plausibility can coexist with failures in spatial memory and scene evolution. The separate analyses and accompanying visual examples help characterize these failures beyond a single aggregate quality score, even though the reliability of the automatic judgments needs further validation.

Critical weaknesses

The experiments do not establish the additional diagnostic benefit of adaptive exploration over predefined trajectories. There is no controlled comparison using the same scenes, models, generation budgets, and scoring procedure. The extended rollout in Section 4.4 demonstrates that the agent can keep operating, but does not establish its reliability across the benchmark or show that adaptation exposes more failures. The observed inconsistencies are useful evidence, but they could also arise under sufficiently long scripted exploration

Questions for authors

Could you compare adaptive exploration with predefined trajectories on a representative subset, keeping initial scenes, models, generation budgets, and scoring consistent? I would be interested in whether adaptation improves valid task completion or increases the number and variety of verified world-model failures.

Reviewer_WghMOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Claiming that the performance of video world models under long-horizon exploration with extensive camera motion remains under-explored, the authors propose WorldProbe, a diagnostic benchmark that uses a goal-conditioned multimodal agent to adapt camera actions online during generation. The evaluation across four dimensions reveals that world models nowadays can often maintain reasonable visual quality for longer, but they still have large problems with long-range consistency.

Critical strengths
  1. The idea of using agentic probe for adaptive camera control and exploration is very interesting and meaningful. Most existing evaluations are short-horizon and use fixed camera trajectories. The authors propose a paradigm where an agent is used to adapt the camera control based on the current generated scene in a closed-loop manner to support evaluation on long-horizon rollouts.

  2. Interesting and meaningful takeaway messages from evaluation. Evaluation on 10 world models shows that current world models can retain decent quality in a long horizon but still lack persistent world consistency. The examples of failed revisits and incorrect object show a common failure mode for current open-source and closed-source video-based world models.

Critical weaknesses
  1. Filtering and retrying rollouts may bias the long-horizon evaluation. The paper filters unsuccessful or invalid rollouts before scoring. However, the failed rollouts or in other word, early termination may be considered as a model failure and should somehow be counted towards the final evaluation score.

  2. Long-range metrics are not sufficiently validated. The metrics finally use VLM to score scene, layout, entity, and texture consistency, etc, which is not very consistent and robust for even sota VLM to evaluate. I am concerned that further human study or other validations are needed to make the main evaluation targets more convencing.

Questions for authors
  1. What is the relation between the "more than 1600 rollouts" in the paper vs. 1201 rollouts in figure 1?

  2. How would the model ranking change if early failures are counted toward the final evaluation score?

  3. How often does ViPE fail or produce unreliable camera poses?

Reviewer_KcEmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

WorldProbe evaluates video world models through goal-conditioned, closed-loop camera control. A multimodal agent uses recent observations, target tracking, and local geometry to choose actions across four types of spatial exploration tasks. The benchmark contains 164 test cases and evaluates ten models, with rollout budgets of up to 180 seconds. The evaluation separates local visual and content quality from spatial memory and evolution consistency. The reported results illustrate how plausible individual frames can coexist with inconsistent scenes and objects over longer interactions. The main contributions are the probing protocol, the implementation across different model interfaces, and the resulting empirical study.

Critical strengths
  1. I find the evaluation setting worthwhile. A predefined camera trajectory can become uninformative when a generated scene introduces an obstacle or changes its layout. Adapting subsequent actions to the observations offers a useful way to continue exploring these outputs. The four task families give this idea a concrete scope, covering constrained navigation, traversal, viewpoint changes, and exploration of new regions.

  2. Getting ten models to accept new actions between generation chunks is useful engineering work. Appendix B provides concrete details about the different inference paths, caches, and decoder states. This implementation effort could make subsequent studies of interactive generation considerably easier. The distinction between local quality and consistency across observations is also appropriate for the questions the benchmark aims to answer.

Critical weaknesses
  1. It is unclear which outputs the reported comparisons represent. The abstract reports more than 1,600 rollouts, Figure 1 lists 1,201, and Table 6 states that the local-quality results use 327 rollouts with complete scores. These may correspond to different processing stages, but I could not find their relationship or the sample counts for each model. More importantly, Section 4.1 and Appendix B.4 describe retrying short rollouts, retaining the longest attempt, and screening out severe visual or trajectory anomalies before scoring. Some of these anomalies may be precisely the failures this benchmark is intended to measure. Table 2 reports pass rates, but does not adequately connect them to the samples used in the subsequent tables. Without this accounting, it is difficult to judge whether the quality comparisons and degradation trends are robust to the selection procedure.

  2. The memory and evolution scores need independent validation. Section 3.3 describes the evaluation dimensions, but the scoring VLM, prompts, and rating criteria are not specified sufficiently to reproduce the scores. I also could not find a comparison with independent human judgments. Memory evaluation introduces an additional concern: the frames to be compared are selected using poses reconstructed from the generated video. If the scene has already changed substantially, errors in pose estimation or matching could prevent a failed revisit from entering the evaluation. Table 5 reports viewpoints per sequence, but not the fraction of rollouts without a valid comparison or how those cases are handled. Several qualitative failures are convincing; they do not establish that the numerical scores reliably distinguish the models.

These are the main reasons for my recommendation. The protocol has potential, but the usefulness of a benchmark depends on knowing what its scores measure and which outputs they cover. Clarifying the sample selection and providing a focused validation of the metrics could change my assessment.

Questions for authors
  1. Could you reconcile the counts of more than 1,600, 1,201, and 327 rollouts, and provide the numbers attempted, retained, and scored for each model? In particular, how often does screening exclude a model failure rather than a controller or execution error, and how are those failures reflected in the reported results?

  2. Which VLM, prompts, and scoring criteria are used for content, memory, and evolution evaluation? Is there any independent validation of these scores? A blind assessment of a representative subset of the existing outputs would help determine whether the reported differences agree with human judgments.

  3. What fraction of rollouts has no valid memory comparison, and how are those rollouts handled in Table 5? Have the selected frame groups been checked to verify that they depict comparable revisited views, including cases with substantial scene or geometry changes?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.