- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces CineScriptBench, a movie-level benchmark that tests whether multimodal LLMs can keep a narrative consistent across a full-length film. Models generate timestamped, scene-level scripts segment by segment while carrying a rolling history forward. The benchmark is built with an automated five-stage pipeline plus human checking, and scored on temporal localization and content accuracy (character, event, scene, history) using an LLM judge. Experiments on proprietary and open-source models show low overall performance. The authors also run ablations on segment length, history format, and ground-truth boundaries, and validate the judge against human labels.
Critical strengths
Well-motivated new benchmark on an under-explored problem. The authors curate a new benchmark to study inter-segment narrative continuity (cross-scene information transfer, character consistency, foreshadowing) in full-length movies, which existing clip-level benchmarks overlook. This matters because it targets a real gap between short-clip understanding and genuine movie-level comprehension, and the progressive script-generation protocol with rolling history is a sensible way to measure it.
Broad evaluation and useful diagnostic analysis. The authors test advanced proprietary and open-source models, such as Qwen, Gemini, MiniMax and GLM. Beyond these scores, they run informative ablations on segment duration, history format, structured output, and ground-truth boundaries, and they meta-evaluate the judge against human labels. This supports their claim that the bottleneck is long-range narrative comprehension rather than temporal localization, and it makes the benchmark credible and actionable for future work.
Critical weaknesses
No methodological contribution beyond evaluation. This paper only proposes a benchmark to evaluate existing approaches, without applying the observed insights to improve models. No training is conducted, so it is unclear whether the findings (for example, the benefit of structured history) translate into actual gains, which limits the paper’s impact beyond diagnosis.
Missing agent-based baselines. Given the limits of model context length and the complexity of film-level long videos, the authors are recommended to also test agent-based approaches. Rather than one-pass model evaluation, agentic models that use tools to selectively read video frames, save history information, and organize memory may perform much better on this task. Without them, the claim that current MLLMs fundamentally struggle may partly reflect the evaluation setup (context-bound, single-pass) rather than a true capability ceiling.
Insufficient comparison with existing long-video benchmarks. Many long-video benchmarks have been proposed, such as LVBench, LongVideoBench, and Video-MME. The authors are recommended to add a table in the paper with a side-by-side comparison between CineScriptBench and existing benchmarks (video length, task format, annotation type, history/cross-segment dependency, etc.) to clearly emphasize the novelty and contribution of the new benchmark.
Questions for authors
Please refer to the weakness section.