acceptodds
Under review as a conference paper at ICLR 2027

ChronoSpatial: A Procedural Video Benchmark for Memory and World-State Reasoning

Abstract

Understanding a video requires maintaining a changing world state: an object may move, disappear behind an obstacle, change hands, or be revisited from a different viewpoint. We introduce ChronoSpatial, a procedural benchmark for evaluating these abilities through delayed, four-choice questions about rendered event sequences. The initial collection contains 215 videos spanning sixteen scenario families, including household activities, navigation, object transfers, and causal interactions. Executable scene descriptions provide reproducible generation and simulator-derived answers. An isolated evaluation harness exposes only anonymous videos and questions, supporting both direct vision-language inference and an agent that inspects video through a bounded visual tool. On the 195-item evaluation split, the GPT-6 Astra agent achieves 79.0%, 96.9%, and 97.4% accuracy with budgets of 16, 64, and 256 inspected frames. Two completed open-model baselines, MiniCPM-V-4.6 and GLM-4.6V-Flash, achieve 28.7% and 25.6% at 64 sampled frames under a separate direct-inference protocol. These initial results motivate closer examination of evidence access, response handling, and sequential complexity. ChronoSpatial provides a reproducible platform for studying visual state tracking while making differences in inference interfaces and computational budgets explicit.

Primary area:
datasets and benchmarks
Keywords:
Video understanding, spatiotemporal reasoning, visual memory, world-state tracking, procedural dataset generation, multimodal evaluation, viewpoint invariance, object permanence.

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ma8YOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

ChronoSpatial is a procedural benchmark of 500 Blender-rendered videos with executable scene histories and programmatically computed answers to four-choice questions. The paper evaluates seven open-weight video models and a budgeted Astra inspection agent, and reports matched input controls, representation and state diagnostics, and exploratory fine-tuning experiments.

Critical strengths

Blender-based synthesis is a valid approach to controlling events and viewpoints in video-understanding evaluation. Executable histories provide reference object states and event sequences that can support controlled diagnostic experiments.

Critical weaknesses

The experimental study is substantially underdeveloped relative to its stated goals.

  1. Limited use of controllability. The generator controls object states and event histories, but the evaluation lacks targeted comparisons that independently vary retention delay, intervening updates, identity binding, or viewpoint. The intended memory and state-tracking capabilities therefore remain insufficiently diagnosed.

  2. Fragmented evaluation design. The five cohorts differ markedly in size (195/12/100/100/61) and task composition, without a sufficiently clear rationale linking them to distinct diagnostic objectives. The unsuccessful level-2/4/6 selection procedure occupies the main text without establishing a calibrated difficulty scale.

  3. Limited explanatory insight. Sections 5-6 largely report score differences without isolating their causes. Comparing direct open-model inference with an interactive Astra scaffold confounds model capability with evidence selection, revisits, and computation. The input controls also do not establish a robust advantage of chronological over shuffled video.

  4. Inconclusive fine-tuning evidence. Small held-out sets (56 test and 34 transfer items), one training seed, and reuse of test histories limit the conclusions. Extreme answer-position bias (89/90 B predictions initially; 75/90 C after balancing) and weak video/no-video separation make it difficult to attribute the observed changes to improved visual state tracking.

Questions for authors
  1. Which existing matched comparisons demonstrate a specific dependence on object-state history or event order, beyond the availability of additional visual information?

  2. Can the open-model/Astra gap be assessed under a shared frame-selection and replay protocol with comparable inference budgets?

  3. After accounting for answer-position bias, what evidence supports a fine-tuning benefit on fresh histories and withheld task families?

Reviewer_NPmHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ChronoSpatial, a benchmark of 500 procedurally generated videos with executable event histories and deterministic answers, designed to evaluate object tracking and historical-state reasoning in video models. Direct video models achieve only 25.1%–35.9% accuracy on the original evaluation set and 24.6%–39.3% on Final-61. In contrast, Astra, an agent that can actively inspect and revisit earlier frames, achieves near-perfect or perfect accuracy across all five evaluation cohorts under the highest budget.

Critical strengths

The paper conducts extensive experiments with multiple open-source models and closed-source agents across varying observation budgets and input conditions, providing strong empirical support for its comparisons.

The experimental setup, evaluation protocol, and resource budgets are described in detail, which enhances transparency and reproducibility.

Critical weaknesses

The motivation for the research problem and the rationale behind the benchmark design are not sufficiently clear.

Although several related benchmarks are discussed, the paper lacks a systematic comparison of task coverage, historical query complexity, controllability, and diagnostic capability. Further empirical analysis is needed to establish the benchmark’s distinct value and necessity.

The evaluation relies mainly on final-answer accuracy, and the reported resource statistics and partial state diagnostics do not fully explain the performance differences. Additional evaluation of event localization, identity tracking, and state updates, under comparable evidence access and resource budgets, would improve the benchmark’s discriminative and diagnostic power.

Questions for authors

please see weakness

Reviewer_aTfWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ChronoSpatial, a benchmark of 500 stylized, procedurally rendered videos generated from executable scene histories. Each video has a four-choice question whose answer is computed from the underlying event history. The evaluation covers five separate cohorts and compares seven open-weight video models, which receive uniformly sampled frames, with an agent that sees the question before adaptively requesting timestamped views under frame and tool-call budgets. The paper also reports matched input controls, explicit state-output probes, external-record diagnostics, and small fine-tuning pilots.

Critical strengths

1, The scene plans specify persistent identities, timed events, camera settings, and deterministic answer programs (Sections 3.1–3.2; Figure 1). This could support precise interventions on historical state, beyond reporting answer accuracy alone.

2, Several experiment results are interesting. The paper reports cohorts separately, includes question-only, single-frame, and shuffled-frame controls, and documents statistical uncertainty and output-finalization costs (Sections 4–5; Appendices B, E, and J). It also explicitly acknowledges that the agent and direct-model interfaces are not equal-compute comparisons.

Critical weaknesses
  1. The synthetic setting needs to be foregrounded when defining the paper’s scope. The title says “procedural,” and the abstract and methods do describe the rendered data. Nevertheless, the introduction begins with a naturalistic everyday scenario, while the implications of evaluating only stylized synthetic videos become clear gradually. This creates a risk that readers interpret the reported scores as evidence about state tracking in natural videos. The present experiments do not test transfer to real footage, where appearance variation, ambiguous actions, occlusion, and scene clutter may change the problem substantially. I recommend stating in the opening paragraph that all 500 videos are synthetic, explaining why this control is useful, and specifying that the results do not estimate performance on natural videos.

  2. The evidence for temporal state tracking is mixed. On the original 64-item matched-control subset, chronological video does not outperform the question-only, middle-frame, or shuffled conditions for the three tested models; none of the nine adjusted paired tests is significant (Table 8; Appendix E). For example, Qwen3-VL-8B-Instruct scores 11/64 with uniform chronological video, versus 16/64 with the question alone and 19/64 with the middle frame. These results do not prove that video is unhelpful, particularly given the small subset and low accuracy, but they limit a strong interpretation of the original benchmark as measuring successful use of temporal evidence. The later cohorts contain selected positive video comparisons—for example, Qwen3.5-9B scores 21/40 versus 8/40 with question-only input on Diverse—but this benefit is not consistent across models and controls (Section 5; Table 16). I would like the central claims to distinguish more sharply between questions designed to require history and empirical evidence that the tested models use video history to answer them.

  3. The agent results measure adaptive inspection, with limited evidence about memory after viewing. The direct models receive 64 uniformly sampled frames. The agent sees the question before inspection and can select timestamps, revisit frames, and crop images, with a different reasoning implementation and joint frame/call budgets (Section 4.1). Consequently, the large original-cohort difference between the agent at the 64-frame allowance (189/195) and the direct models (49–70/195) does not isolate an effect of memory capacity or model architecture (Table 2). The paper acknowledges this, but the contrast remains central to its presentation.

  4. Missing discussion of very related works. Several previous video reasoning benchmark works, including VideoReasonBench and VideoZeroBench, are missing.

Questions for authors

1, Can the authors quantify human answer accuracy, rather than only perceived solvability, on a randomly selected subset under clearly specified viewing and replay conditions?

2, Can the executable plans be used for a small, pre-specified set of matched interventions—for example, changing one decisive event, with camera-only and irrelevant-event changes as controls—and can model answers be tested for the corresponding state change?

3, Within the agent interface, what happens when question timing, adaptive versus fixed frame selection, and frame versus call allowances are varied separately?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.