acceptodds
Withdrawn from ICLR 2027

Beyond Isolated Clips: Benchmarking Inter-Segment Narrative Continuity in Movie-Level Video Understanding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent advancements in multimodal large language models (MLLMs) have extended video understanding from short clips to long-form cinematic content. However, existing benchmarks predominantly evaluate isolated video segments, overlooking the narrative continuity essential to full-length movies, such as cross-scene information transfer, character consistency, and plot foreshadowing. To bridge this gap, we introduce CineScriptBench, the first movie-level benchmark explicitly designed to evaluate inter-segment historical consistency in cinematic script generation. Comprising 73 full-length movies (153 hours), 5,524 scenes, and 56,993 fine-grained evaluation rubrics, CineScriptBench is constructed via a novel five-stage automated pipeline that injects rolling historical context into scene-level annotations. We propose a comprehensive evaluation protocol measuring both temporal localization and content accuracy, featuring a dedicated dimension for historical consistency via an LLM-as-a-Judge framework. Extensive evaluations of frontier MLLMs reveal severe limitations in long-form narrative understanding. Our findings highlight the critical gap between current MLLM capabilities and true movie-level comprehension, establishing a rigorous baseline for future research in cinematic narrative reasoning.

Primary area:
datasets and benchmarks
Keywords:
Multimodal Large Language Models, Long-Form Video Understanding, Cinematic Script Generation, Narrative Continuity, Evaluation Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_YbwPOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CineScriptBench, a movie-level benchmark that tests whether multimodal LLMs can keep a narrative consistent across a full-length film. Models generate timestamped, scene-level scripts segment by segment while carrying a rolling history forward. The benchmark is built with an automated five-stage pipeline plus human checking, and scored on temporal localization and content accuracy (character, event, scene, history) using an LLM judge. Experiments on proprietary and open-source models show low overall performance. The authors also run ablations on segment length, history format, and ground-truth boundaries, and validate the judge against human labels.

Critical strengths

Well-motivated new benchmark on an under-explored problem. The authors curate a new benchmark to study inter-segment narrative continuity (cross-scene information transfer, character consistency, foreshadowing) in full-length movies, which existing clip-level benchmarks overlook. This matters because it targets a real gap between short-clip understanding and genuine movie-level comprehension, and the progressive script-generation protocol with rolling history is a sensible way to measure it.

Broad evaluation and useful diagnostic analysis. The authors test advanced proprietary and open-source models, such as Qwen, Gemini, MiniMax and GLM. Beyond these scores, they run informative ablations on segment duration, history format, structured output, and ground-truth boundaries, and they meta-evaluate the judge against human labels. This supports their claim that the bottleneck is long-range narrative comprehension rather than temporal localization, and it makes the benchmark credible and actionable for future work.

Critical weaknesses

No methodological contribution beyond evaluation. This paper only proposes a benchmark to evaluate existing approaches, without applying the observed insights to improve models. No training is conducted, so it is unclear whether the findings (for example, the benefit of structured history) translate into actual gains, which limits the paper’s impact beyond diagnosis.

Missing agent-based baselines. Given the limits of model context length and the complexity of film-level long videos, the authors are recommended to also test agent-based approaches. Rather than one-pass model evaluation, agentic models that use tools to selectively read video frames, save history information, and organize memory may perform much better on this task. Without them, the claim that current MLLMs fundamentally struggle may partly reflect the evaluation setup (context-bound, single-pass) rather than a true capability ceiling.

Insufficient comparison with existing long-video benchmarks. Many long-video benchmarks have been proposed, such as LVBench, LongVideoBench, and Video-MME. The authors are recommended to add a table in the paper with a side-by-side comparison between CineScriptBench and existing benchmarks (video length, task format, annotation type, history/cross-segment dependency, etc.) to clearly emphasize the novelty and contribution of the new benchmark.

Questions for authors

Please refer to the weakness section.

Reviewer_Q2YZOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CineScriptBench to address the limited attention to inter-segment narrative continuity in existing long-video benchmarks. It evaluates movie-level script generation and historical consistency using 73 full-length movies, 5,524 scenes, and 56,993 fine-grained rubrics. A five-stage pipeline constructs scene annotations and historical information, while evaluation measures both temporal localization and content performance. The best Omni model achieves an overall content score of 31.06% on the full test set, and all models score below 10% on History in the main results, highlighting limitations in current movie-level narrative understanding.

Critical strengths
  1. The research question is meaningful. Existing long-video benchmarks often use question answering, while script-generation evaluations mainly focus on isolated clips. Incorporating cross-scene historical information and narrative continuity into movie-level generation addresses an underexplored evaluation dimension.

  2. The experiments and diagnostic analyses are extensive. Beyond overall model comparisons, the paper examines segment duration, history organization, structured output, and ground-truth scene boundaries. Content scores remain low even with ground-truth boundaries, showing that improving temporal localization alone is insufficient to solve movie-level script generation.

Critical weaknesses
  1. The contributions of pretrained movie knowledge and inter-segment reasoning are not sufficiently distinguished. The benchmark includes well-known movies such as Interstellar, for which models may already possess character, plot, and background knowledge. Although the paper includes a no-history ablation, it lacks controls that directly examine movie familiarity. It remains unclear how much of the performance comes from accumulating information across segments rather than visual cues activating existing movie knowledge.

  2. Data construction and judge selection may introduce biases associated with model provenance. The construction pipeline uses Gemini-3.1-Pro-Preview, while content evaluation uses Qwen-3.8-27B; both are also evaluated as candidate models. Although several judges are compared against human labels, the paper does not report scoring errors by the generating model's family or ranking stability under alternative judges. The final judge also participates in rubric self-checking, and agreement between reference scripts and rubrics does not replace independent verification of their factual grounding.

  3. There are implementation gaps between the evaluation protocol and the published prompts. The overlap mechanism in Algorithm 1 depends on predicted end times exceeding the current window and passes overlapping content to the next window. However, Card E1 restricts timestamps to candidate boundaries within the current window and provides no separate overlap-context input. The current PDF also omits the actual judge prompt and the structured scene-output variant, while the section references for history-organization variants do not match the main text. These omissions limit reproducibility.

  4. A human baseline is missing, and agreement in human annotation and evaluation is insufficiently reported. The paper provides no human script-generation baseline under the same task conditions, leaving the achievable performance ceiling, task solvability, and model–human gap unclear. Without this reference, it is difficult to determine whether the low model scores primarily reflect task difficulty or problems in task design, reference annotations, or scoring rubrics. Moreover, although the paper reports 92.5% agreement between the judge and human labels, it does not adequately explain how consistently different human annotators agree during dataset annotation and evaluation, or how disagreements are resolved.

  5. Some comparisons mix differences in input conditions and evaluation subsets. Open-source Omni models use 0.4 FPS, whereas other models generally use 2 FPS. Their very low content scores may reflect missing visual information, context limits, and output-format failures, but these factors are not disentangled. Moreover, Omni models are evaluated on all 73 movies, whereas VL models use only the 17-movie subtitle subset. These results alone do not establish that visual subtitle information compensates for missing audio. In the history-format comparison, Timeline preserves and appends existing entries while Plain can compress and rewrite them, so differences in history length and retained information may also affect the results.

  6. There are omissions in figure disclosure and benchmark naming. Figure 1 uses AI-generated imagery, but the AI Use Statement discloses only language polishing. The caption of Table 3 also incorrectly names the benchmark OMNIMOVIEBENCH and should be corrected.

Questions for authors
  1. How is the overlap context in Algorithm 1 supplied during actual evaluation? Since Card E1 restricts timestamps to the current window, how can the model trigger detection of a scene extending beyond it? Please provide the corresponding implementation and judge prompt, and clarify unmatched-scene handling, matching constraints, and content-score aggregation.

  2. Could the authors report results for representative Omni models on the same 17 subtitled movies, together with a within-model ablation that removes audio? To investigate the low open-source Omni scores, could they report format-failure rates and evaluate a representative strong model at 0.4 FPS to distinguish input-information effects from model capability?

  3. How does History scoring distinguish omitted background facts from actual narrative contradictions? Could the authors verify the preceding-scene evidence for historical rubrics on a small subset and provide human-generation scores under the same output constraints? Please also report the number of human annotators, the proportion of annotations revised, and inter-annotator agreement.

  4. Could movie-familiarity groups or suitable controls quantify the influence of existing movie knowledge? Also, do history-format comparisons control the history token budget and truncation rules, and how much would History scores change when verified histories are supplied instead of model-generated histories?

Reviewer_sZQcOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces CineScriptBench, a benchmark for progressive script generation over full-length movies that emphasizes temporal localization and cross-segment historical consistency. It contains 73 movies selected from Douban, 5,524 annotated scenes, and 56,993 evaluation rubrics, with the annotations generated by Gemini-3.1-pro-preview. Model outputs are scored by Qwen-3.8-27B, which is selected as the judge through a meta-evaluation against sampled human annotations, and a range of multimodal models are evaluated with temporal accuracy and content accuracy reported separately.

Critical strengths

The paper targets a relevant direction, evaluating detailed descriptions of movie content within their broader narrative context rather than as isolated clips. Full-length movies require models to keep track of characters and events across distant parts of the story, which short-clip benchmarks cannot assess. This is the main consideration in favor of the paper, since a reliable benchmark of this kind would be useful for evaluating long-form video understanding.

Critical weaknesses
  1. The reliability of the annotations and the evaluation is not established. The scene annotations and rubrics are generated by Gemini-3.1-pro-preview, and no human verification of these annotations is reported. Because the judge, Qwen-3.8-27B, compares model outputs with these references using text alone, the evaluation cannot establish whether the references are grounded in the audiovisual content of the movies. Both models are also among the evaluated models, but the potential bias from this overlap is not analyzed. The judge meta-evaluation in Table 2 compares judges with sampled human annotations, yet the annotators, the annotation protocol, and inter-annotator agreement are not described. In addition, the vision-language models in Table 1 are evaluated on different numbers of samples, which makes their results difficult to compare. The paper also does not state whether the benchmark data will be released. This is consequential because the value of a benchmark depends on reliable references and judging, and on its availability to the community.

  2. The benchmark does not show that it measures long-range narrative understanding rather than local description accuracy. Historical consistency, a central notion of the benchmark, is not clearly defined, and the history analysis in Section 4.6 varies only the textual format of the history rather than its presence or the distance of the dependencies, so it offers little insight into consistency. No controlled history ablation or analysis by dependency distance is provided. The paper also does not relate the benchmark to existing movie-level video understanding tasks, so it is unclear what it adds beyond dense captioning and long-video understanding benchmarks. This is consequential because long-range narrative understanding is the main capability the benchmark claims to evaluate. Addressing this would require new analyses and benchmark positioning rather than clarification alone.

Questions for authors
  1. Was any portion of the generated annotations and rubrics verified by humans against the movies? If so, what were the procedure and the observed error rate?
  2. Who were the human annotators in the judge meta-evaluation (Table 2), what protocol did they follow, and what was the inter-annotator agreement?
  3. Gemini-3.1-pro-preview generated the references, yet it does not achieve the best score on several items. How do the authors explain this, and how does using the same models for annotation, judging, and evaluation affect the rankings?
  4. How is historical consistency defined, and how do the results change when the history is removed or when dependencies span longer distances?
  5. Why were these models selected for evaluation, and why are the vision-language models in Table 1 evaluated on different numbers of samples?
  6. How are shot, scene, and segment defined, and which of them is the unit of evaluation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.