- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces StateBench, a 200-case benchmark for world-state reasoning across video segments, covering past-visible states, occluded processes, and complex transitions. Its evaluation separates completion of the requested action from state correctness and penalizes exposure failures in an all-case score. The paper also proposes StateAgent, which parses and updates structured entity states, grounds them into a future frame through image editing, and generates the continuation. Experiments report an improvement in the all-case score from 45.4 to 69.3, with additional evaluation on minute-long storytelling.
Critical strengths
-
A focused and consequential evaluation target. The benchmark tests whether continuations preserve states established before a segment boundary, addressing an important limitation that appearance consistency alone does not capture.
-
Exposure-aware evaluation. Separating action completion from state correctness helps distinguish failures to reveal the state from incorrect revealed states. The all-case score prevents high conditional scores from masking exposure failures.
-
A constructive baseline with substantial reported gains. StateAgent provides inspectable intermediate representations and improves the all-case score from 45.4 to 69.3. Comparisons with direct frame editing and textual state conditioning provide useful evidence for the proposed design.
Critical weaknesses
-
Insufficient documentation of benchmark construction and reference-state validity. The provided version does not adequately specify the provenance, selection, and validation of historical videos. It is therefore difficult to assess whether the reference states are clearly supported by the actual videos, particularly for occluded or complex transitions.
-
Insufficient evaluator specification and human calibration. The evaluation model, prompts, and sampling settings are insufficiently documented. Agreement with human judgments is not reported, limiting the reproducibility and reliability of the scores.
-
Limited evidence for long-horizon benefits and result stability. The Bench consistency gain is small (0.586 to 0.594), and its connection to sustained physical-state consistency remains unclear. Repeated-run results or uncertainty estimates are missing, leaving the robustness of the long-horizon gains uncertain.
Questions for authors
see above