- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
ChronoSpatial is a procedural benchmark of 500 Blender-rendered videos with executable scene histories and programmatically computed answers to four-choice questions. The paper evaluates seven open-weight video models and a budgeted Astra inspection agent, and reports matched input controls, representation and state diagnostics, and exploratory fine-tuning experiments.
Critical strengths
Blender-based synthesis is a valid approach to controlling events and viewpoints in video-understanding evaluation. Executable histories provide reference object states and event sequences that can support controlled diagnostic experiments.
Critical weaknesses
The experimental study is substantially underdeveloped relative to its stated goals.
-
Limited use of controllability. The generator controls object states and event histories, but the evaluation lacks targeted comparisons that independently vary retention delay, intervening updates, identity binding, or viewpoint. The intended memory and state-tracking capabilities therefore remain insufficiently diagnosed.
-
Fragmented evaluation design. The five cohorts differ markedly in size (195/12/100/100/61) and task composition, without a sufficiently clear rationale linking them to distinct diagnostic objectives. The unsuccessful level-2/4/6 selection procedure occupies the main text without establishing a calibrated difficulty scale.
-
Limited explanatory insight. Sections 5-6 largely report score differences without isolating their causes. Comparing direct open-model inference with an interactive Astra scaffold confounds model capability with evidence selection, revisits, and computation. The input controls also do not establish a robust advantage of chronological over shuffled video.
-
Inconclusive fine-tuning evidence. Small held-out sets (56 test and 34 transfer items), one training seed, and reuse of test histories limit the conclusions. Extreme answer-position bias (89/90 B predictions initially; 75/90 C after balancing) and weak video/no-video separation make it difficult to attribute the observed changes to improved visual state tracking.
Questions for authors
-
Which existing matched comparisons demonstrate a specific dependence on object-state history or event order, beyond the availability of additional visual information?
-
Can the open-model/Astra gap be assessed under a shared frame-selection and replay protocol with comparable inference budgets?
-
After accounting for answer-position bias, what evidence supports a fine-tuning benefit on fresh histories and withheld task families?