acceptodds
Under review as a conference paper at ICLR 2027

CINEMA-Bench: Benchmarking Cinematic Expressiveness Capabilities of Multi-Shot Video Generation

Abstract

Recent advances in video generation have moved toward multi-shot creation, approaching professional video production. However, the field still lacks a clear definition of cinematic expression capabilities and quantitative evaluation protocols to benchmark the progress. To fill this gap, we present CINEMA-Bench (CINematic Expressiveness in Multi-shot Video Assessment), a reference-grounded benchmark for evaluating the cinematic expression of multi-shot video generation models. Inspired by established film theory, we propose a hierarchical ”Vocabulary–Grammar–Rhythm” taxonomy spanning three levels and 21 dimensions, covering shot-level camera language, inter-shot continuity grammar, and sequence-level temporal perception. We construct a multi-shot hierarchical prompt suite derived from real-world films and TV series across Mikels’ eight emotions. We further design automatic metrics validated by human alignment to assess cinematic composition from the Director’s Perspective, and complement them with user studies from the Audience’s Perspective. Systematic evaluation of six state-of-the-art multi-shot video generation models reveals that, despite reasonable use of cinematic elements, current methods still exhibit notable gaps in inter-shot continuity, rhythm and emotional delivery, shot-count controllability, and audience perception, offering actionable insights for cinematically expressive video generation. This work presents an early exploration toward a comprehensive definition and quantification of cinematic expressiveness, bridging artistic expressiveness and computational evaluation.

Primary area:
datasets and benchmarks
Keywords:
Video Benchmark, Multi-Shot Video, Video Quality Evaluation, AIGC Videos, Cinematography;Computational Creativity

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ebCqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Current video generation benchmarks primarily focus on temporal coherence, physical fidelity and semantic alignment, while overlooking the artistic expressiveness of multi-shot filmmaking, which is very important in current art creativity. To address this gap, CINEMA-Bench constructs high-quality prompts from real, professionally produced film clips and evaluates six leading video generation methods using a taxonomy grounded in film theory. This work represents an early exploration toward a comprehensive definition and quantification of cinematic expressiveness, bridging artistic expression and computational evaluation.

Critical strengths

It examines multi-shot video generation from the distinctive perspective of artistic expressiveness.

The prompts cover Mikels’ eight emotion categories, and the evaluation taxonomy is grounded in film theory, providing a well-founded methodological basis.

Critical weaknesses

In Table 4, cinematic diversity and rhythm rationality show limited agreement with human judgments. Many of the paper’s automatic metrics rely on Gemini, which interprets videos through sampled frames. Whether this approach can yield an understanding of cinematic content comparable to that of human viewers remains to be established.

Questions for authors
  1. Cinematic expression is closely intertwined with music and sound. Could CINEMA-Bench’s exclusion of audio lead to a biased assessment of artistic quality? I think a comprehensive evaluation should be multimodal.

  2. Could the evaluation metrics in CINEMA-Bench be used as reward signals to train video generation models?

  3. As our understanding of art and cinematic conventions evolves, could this evaluation framework become outdated? How could it adapt to such changes?

  4. Could you include new models like LTX in the model evaluation?

  5. Different models may respond differently to prompt styles, especially descriptions of shot transitions. Were the prompts tailored to each model during evaluation? If not, could the observed performance differences partly reflect biases arising from model-specific prompt preferences rather than differences in cinematic capabilities? How to seperate this bias?

Reviewer_XG8xOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents CINEMA-Bench, a benchmark for evaluating the cinematic expressiveness of multi-shot video generation models. Drawing from film theory, it proposes a hierarchical "Vocabulary–Grammar–Rhythm" taxonomy with 21 dimensions covering shot-level camera language (9 dims), inter-shot continuity grammar (6 dims), and sequence-level temporal perception (6 dims, split into Director's and Audience's perspectives). The benchmark contains 300 multi-shot prompts derived from real films/TV across Mikels' 8 emotion categories, with hierarchical metadata (global caption, shot-level captions, cinematic labels). Six SOTA multi-shot video generation models are evaluated, revealing gaps in inter-shot continuity, rhythm-emotion delivery, shot-count controllability, and audience perception. Automatic metrics are validated against human annotations.

Critical strengths
  1. The Vocabulary–Grammar–Rhythm hierarchy is clearly derived from established film theory and covers the full spectrum from per-shot elements to sequence-level perception. The 21-dimension structure is comprehensive yet interpretable, and the dual Director/Audience perspectives at the Rhythm level are a thoughtful design. This taxonomy itself is a useful contribution that future work can build upon regardless of the specific benchmark implementation.

  2. Unlike prior benchmarks that rely on LLM-generated prompts, CINEMA-Bench derives its 300 prompts from real films/TV with emotion-balanced coverage (Mikels' 8 categories). This grounds evaluation in professional practice and enables meaningful comparison between generated and real cinematic content. The hierarchical metadata (global + shot-level + cinematic labels) is practically useful for structured evaluation.

Critical weaknesses
  1. Weak human alignment on the most important metrics severely undermines the benchmark's reliability. Table 4 reveals that the automatic metrics for sequence-level dimensions — which are the paper's most novel contribution — correlate poorly with human judgments.

  2. The benchmark evaluates but does not advance — limited methodological or analytical contribution beyond taxonomy proposal.

Questions for authors
  1. Given the weak human alignment on rhythm rationality (ρ=34.9%) and cinematic diversity (ρ=37.4%), how confident are you in the rhythm and diversity rankings reported in Table 1? Would the model ranking change if human scores were used instead of automatic metrics for these dimensions?

  2. The prompt suite is derived from 300 real videos. How were these 300 selected from the larger pool? Is there selection bias toward certain genres, production styles, or time periods that could affect the emotion distribution?

  3. For the Audience's Perspective dimensions (narrative comprehension, cognitive engagement, emotional resonance), how many human raters were used per video, and what was the total annotation cost? These are highly subjective — is the sample size sufficient for reliable conclusions?

  4. Several typos suggest limited proofreading ("cinmeatic" line 192, "experssiveness" line 1337, "insufficeny" line 274, "machness" line 706, "preception" line 188). Can the authors confirm this is the intended submission version?

1 comment

Since the authors did not submit a rebuttal, I will maintain my original score.

Reviewer_oQFBOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CINEMA-Bench, a benchmark for evaluating cinematic expressiveness in multi-shot video generation. It contains 300 sequences derived from films and TV series, each with 3–10 shots and one of eight Mikels emotion categories. The proposed Vocabulary–Grammar–Rhythm taxonomy contains 21 dimensions covering shot-level camera language, transition-level continuity, sequence-level composition, and audience perception. Most Director’s Perspective scores are produced automatically using Gemini-2.5-Pro labels or hand-designed visual metrics, while two film experts provide human validation and Audience’s Perspective ratings. Six generation systems from three model paradigms are evaluated.

Critical strengths
  1. Evaluating whether camera language, editing choices, and sequence rhythm serve a narrative is a useful direction beyond conventional visual-quality and prompt-alignment metrics. The shot, transition, and sequence hierarchy provides an intuitive structure for discussing different sources of cinematic failure.
  2. The suite spans 300 real-film-derived prompts with 3–10 shots and evaluates six systems across pretrained, two-stage, and holistic generation paradigms. This coverage is broader than evaluations restricted to short sequences or a single generation paradigm.
Critical weaknesses
  1. I find the contribution insufficiently distinguished from existing cinematic evaluations. Lines 112–116 claim the first comprehensive benchmark for cinematic expressiveness, but FilmBench and EvalVerse have substantial overlap with the paper’s motivation and scope. FilmBench also uses multi-shot prompts derived from professional films, a cinematic-language taxonomy, and an automatic evaluation protocol. EvalVerse covers multi-shot narrative and spatial continuity, affective progression, shot-duration rationality, and rhythmic layering. CineTechBench has a narrower task, but already evaluates six shot-level dimensions used here: scale, angle, composition, movement, lighting, and color. FilMaster is cited as a generation system, but its FilmEval benchmark is not discussed; FilmEval already evaluates camera language, rhythm and flow, and engagement, and validates its automatic scores against human judgments. The Vocabulary–Grammar–Rhythm organization may be useful, but the paper does not explain what additional evaluation value its taxonomy, data, or protocol provides over these prior benchmarks.
  2. I am not convinced that the sequence-level scores adequately measure the cinematic qualities claimed. The rhythm metric excludes shot duration at lines 324–327 and summarizes adjacent-shot changes through averages and variability. It therefore cannot distinguish sequences that have the same change statistics but place those changes at different points in the story. Lines 450–465 nevertheless describe Table 4 as confirming human alignment, although the Spearman correlations are only 0.374 for cinematic diversity and 0.349 for rhythm rationality. Table 5 also reports 27.62% Top-1 agreement for emotion labeling at lines 729–734 without clearly explaining the comparison protocol. Since the rhythm score is conditioned on the target emotion, the unclear labeling protocol makes the reliability of this score difficult to assess. Finally, Wan, Seedance, and the other methods receive total duration budgets of min⁡(3N,9)\min(3N,9), min⁡(3N,15)\min(3N,15), and 3N3N seconds, respectively, at lines 702–711. The long-video and shot-count results therefore do not isolate model capability from duration and interface constraints. These limitations weaken the interpretation of the sequence-level scores and the principal system comparisons.
Questions for authors
  1. What additional evaluation value does CINEMA-Bench provide beyond FilmBench, EvalVerse, CineTechBench, and FilmEval? In light of these works, what is the basis for the “first comprehensive benchmark” claim at lines 112–116?
  2. What evidence supports the claim of human alignment for cinematic diversity and rhythm rationality, given their Spearman correlations of 0.374 and 0.349 in Table 4? Please also clarify what the 27.62% Top-1 emotion agreement in Table 5 measures and whether the target emotion labels were checked. Appendix A states “The detailed prompts are shown in xx” at lines 672–677, but no corresponding prompts are included. Which exact evaluator prompts were used to produce the reported results?
  3. How much of the differences in Tables 2 and 3 can be attributed to the unequal maximum-duration constraints, and do the long-video and shot-count conclusions remain after accounting for these constraints?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.