acceptodds
Under review as a conference paper at ICLR 2027

SpaceTimeBench: Evaluating Space And Time In The World Model

Abstract

Generative world models aim to serve as universal physical simulators, yet existing benchmarks fail to verify the fundamental prerequisite of Newtonian Law: that space and time must remain absolute and mathematically consistent across different observer perspectives. To bridge this evaluation gap, we introduce SpaceTimeBench, a novel Video-to-Video framework that utilizes prompt-injected geometric anchors alongside DePT3R and SAM 2 to extract strictly normalized 4D coordinates, testing whether models perfectly conserve spatial distances and temporal durations across varied camera views. Extensive evaluations across 17 leading architectures reveal a severe capability deficit quantified by our Final Spacetime Yield (FSY), demonstrating that even state-of-the-art models peak at a mere 65.0% success rate. Crucially, SpaceTimeBench functions as an evaluation oracle to expose a False Discovery Rate (FDR) of up to 42.9% regarding spurious compliance in existing benchmarks, proving that evaluating Newton's First Law without verifying underlying spatiotemporal prerequisites severely inflates scores and masks the reality that current architectures remain mere two-dimensional pattern matchers.

Primary area:
datasets and benchmarks
Keywords:
Generative World Models, Physical Simulation, Video Generation, Benchmarking, Spacetime Invariance, 3D Tracking, Newtonian Mechanics

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_r3j2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SpaceTimeBench evaluates whether video models preserve spatial displacement and temporal progression when generating a broader-view version of an input video. It uses a sphere and a rotating pinwheel as measurement references, applies seven sequential checks, and reports results on 17 models across 350 prompts.

Critical strengths
  1. Cross-video event consistency is a useful evaluation target that complements conventional video-quality assessment.
  2. The staged checks provide useful diagnostics by separating event loss, reference-object failures, and spatial or temporal discrepancies.
Critical weaknesses
  1. The evaluation relies on models generating a broader view of the same event from a text prompt, but it does not clearly verify whether the viewpoint actually changes as intended. Some models may retain a view close to the original, while others may generate substantially different views. This could affect the measured cross-view consistency and make comparisons across models less reliable.

  2. The reliability of the spatial and temporal measurements is unclear. The projected rotation angle of the clock may vary with the viewing direction, even when the underlying rotation is unchanged. In addition, the variance of an object's 3D point cloud depends on its shape and size, not only on tracking uncertainty. These factors may affect the estimated spatial and temporal consistency and lead to incorrect failure diagnoses.

Questions for authors
  1. How do you verify that the generated video shows the intended change in viewpoint? For example, how do you distinguish a genuine change in camera viewpoint from simply zooming out or making the original scene smaller in the frame?

  2. Have you evaluated the pipeline on multi-view videos with known camera parameters, 3D geometry, and temporal information? If so, how often does it incorrectly identify physically consistent videos as failures?

  3. In Table 4, are the baseline-positive samples selected according to WorldModelBench's judgments or the F1–F5 checks in SpaceTimeBench? Since the paper defines FDR based on positive results from WorldModelBench, please clarify how these samples are selected and how the denominator is calculated.

  4. Table 4 extends beyond the right edge of the page. Please resize the table to fit within the page margins.

Reviewer_UzdwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SpaceTimeBench evaluates whether a generated alternative view preserves the spatial and temporal properties of an event. It adds a sphere and a spinning pinwheel to video prompts as spatial and temporal references, then uses DePT3R and SAM 2 to estimate object positions and clock phases. A sequence of checks produces a final success rate across video generation models. The paper also uses these results to argue that existing physics benchmarks accept some physically inconsistent videos.

Critical strengths
  1. Preserving the same event across viewpoints is a useful evaluation target. It addresses a limitation that can be missed when a generated video is assessed only in isolation.

  2. The paper attempts to provide interpretable failure categories and results across different scene types. If the measurements were validated, this breakdown could help explain where models lose consistency.

Critical weaknesses
  1. The measurement procedure needs validation, only after that can its failures be interpreted as physical errors. In Section 3.3, the sphere's visible surface is used to estimate its dimensions, although these need not match the dimensions of the full sphere. The pinwheel's projected angle also depends on viewpoint, and regular rotation within each video does not ensure that both clocks run at the same rate. In Section 3.4, surface-point spread is treated as tracking uncertainty without calibration. There is no reported check on known-correct scenes to show how often these choices reject valid videos. The interpretation is also too broad: the spatial test compares only start-to-end displacement lengths, which cannot establish agreement throughout a trajectory. It supports a restricted consistency test rather than general Newtonian compliance.

  2. The headline false-discovery claim and the reported tables are not adequately supported. Section 5.2 describes errors made by WorldModelBench, but Table 4 defines the positive set using the proposed pipeline's own preliminary checks. The paper does not report WorldModelBench judgments on the same videos or their overlap with the claimed failures. Separately, some supposedly exclusive categories in Table 3 do not sum to the stated total, the Veo Human Activities row sums to 104%. The aggregate results also differ across tables without an explanation. These issues prevent a reliable interpretation of the main quantitative conclusions.

Questions for authors
  1. What existing calibration shows the evaluator's false-failure rate on physically valid videos? How are partial sphere visibility, viewpoint-dependent clock phase, and clock-rate differences handled?
  2. Were WorldModelBench judgments obtained on the same videos? If so, please provide the corresponding positive counts and overlap with the spatial and temporal failures.
  3. Can the discrepancies within Table 3 and across Tables 2 and 4 be reconciled from the existing per-video records? Do the tables use the same samples and evaluation settings?
Reviewer_4qDrOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SpaceTime Bench, which evaluates whether video generative models preserve the space and time consistently when asked to generate from different viewpoints. It augments prompts with a static sphere and spinning pinwheel, so that they can use these as anchors to understand space and time scales. They use DePT3R and SAM2 to extract relevant information for evalution. The results show that 17 models evaluated do not perform well on their benchmakr and existing physical consistency benchmarks have substantial false-positive rates.

Critical strengths
  1. the problem is interesting and relevant.
  2. the design of the benchmark is interesting. The cross-view setup, and the added objects for specific space and time evalutions are interesting.
Critical weaknesses
  1. The accuracy of SAM2 and DePT3R model are not established. It is not clear how good the evaluation models are for the generated samples from different generative models. Are these models generalizable enough?
  2. The insertion of reference objects is an interesting idea, but it also conflates the evaluation score. It is not clear that the evaluated model can re-render these reference objects consistently across different viewpoints. The detected inconsistency could be originated from the imperfect generation of the reference objects or the prerequisite of the physics.
  3. There are a lot hallucinations in the citations. The format of the citations is also incorrect in many of them. For example,

Yann LeCun and Courant. A path towards autonomous machine intelligence. OpenAI, Meta AI Research, 2022.

OpenAI. Worldsimbench: Towards video generation models as world simulators. Yi-Ran Qin, Zhelun Shi, Jiwen Yu, Xijun Wang, Enshen Zhou, Lijun Li, 2024. doi: 10.48550/arXiv.2410.18072.

Peng Liu, Xiaoming Ren, Fengkai Liu, Qinglin Xie, Qibing Zheng, Yingze Zhang, Huchuan Lu, and Yuchao Yang. Dynamic-i2v: Exploring image-to-video generation models via multimodal llm. arXiv.org, 2025. doi: 10.48550/arXiv.2505.19901.

Jing Gu, Xian Liu, Yu Zeng, Ashwin Nagarajan, Fangrui Zhu, Daniel Hong, Yue Fan, Qianqi Yan, KAI-QING Zhou, Ming-Yu Liu, et al. Phyworldbench”: A comprehensive evaluation of physical realism in text-to-video models. arXiv.org, 2025. doi: 10.48550/arXiv.2507.13428. Evaluates object motion, energy conservation, and rigid body interactions.

S. Campbell et al. Newton - a small benchmark for interactive foundation world models. In OpenReview, 2025. Tests object permanence by using interactive camera-conditioned actions (e.g., rotate 90 degrees left, then right) to verify the model remembers hidden geometric locations.

etc.

Questions for authors

See weaknesses.

  1. The V2V requirement in the evaluation introduces additional capabilities of video generative models, and could introduce confounding factors in evaluation. This requirement also limits the video generative models that can be evaluated. To me, the target task can be evaluated within a single video, but comparing differen chunks. Can the authors discuss the possibility of using a single video to conduct the evaluation? What could be difficult if we only use a single video?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.