acceptodds
Withdrawn from ICLR 2027

CineSubBench: Evaluating LLMs on Long-Form Narrative and Cultural Understanding from Multilingual Movie Subtitles

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

33 views

Abstract

Large language models are increasingly evaluated in specialized domains such as law, medicine, software engineering, and cybersecurity, yet film remains comparatively underexplored despite requiring long-form narrative integration, multilingual interpretation, and culturally situated audience judgments. We introduce CineSubBench, a benchmark for evaluating long-context film understanding from multilingual movie subtitles. A subtitle track represents a film as thousands of short, temporally ordered utterances from which models must reconstruct characters, relationships, events, causal progression, and themes without explicit scene or event structure. CineSubBench contains 1,012 films with complete subtitle coverage in six languages, yielding 6,072 tracks and 8.13M timestamped subtitle entries. It provides a matched multi-task, multilingual, and multicultural (MultiX) evaluation setting: seven tasks span narrative reconstruction and abstraction, genre prediction, age suitability, country-specific motion-picture ratings across ten national classification systems, and subtitle-grounded language safety. Across nine LLMs, plot premises are recovered more reliably than event-complete synopses; cross-lingual consistency varies substantially across models and languages; national rating systems expose distinct calibration patterns; and strong profanity is far easier to ground than mild obscenity. CineSubBench establishes film as a long-context LLM evaluation domain and provides a unified benchmark for measuring narrative, multilingual, cultural, and evidence-grounding capabilities.

Primary area:
datasets and benchmarks
Keywords:
Long-Context Understanding, Multilingual Evaluation, Multicultural Evaluation, Movie Subtitles, Narrative Understanding, Cultural Prediction, Film Understanding

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Jro6OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CineSubBench, a benchmark that uses multilingual film subtitles to evaluate large language models (LLMs) on long-context narrative understanding and cross-cultural judgment. The benchmark contains 1,012 films released between 1977 and 2026 and is organized around a multi-task, multilingual, and multicultural (MultiX) setting. The authors evaluate nine closed and open-weight models and use GPT-5.6 Luna as a reference-based, two-stage LLM judge. The results reveal factual errors in narrative generation and performance differences across languages in cultural and safety-related evaluation.

Critical strengths

The dataset has broad scale and multilingual coverage: it integrates multiple sources through IMDb identifiers and covers six subtitle languages—English, Arabic, Indonesian, Persian, Romanian, and Vietnamese—with 1,012 films, 6,072 subtitle tracks, and 8.13M timestamped subtitle entries.

Critical weaknesses
  • The benchmark lacks a human baseline and sufficient reporting of annotation agreement. The paper reports high agreement between independent LLM judges, but does not report human performance under the same subtitle inputs, task definitions, output constraints, and scoring setup. It is therefore unclear how solvable the tasks are, what a reasonable performance ceiling is, or how large the model–human gap may be. Low scores could reflect limited model capability, task difficulty, reference-label limitations, or scoring design, and these possibilities are difficult to distinguish. The paper should also report the number of annotators, independent annotation agreement, and disagreement-resolution procedures for subtitle cleaning, task-label curation, manual audits, and judge evaluation.

  • The impact of pretrained priors and film-identity memorization is not yet quantified. Although the standard evaluation hides film titles, release years, and cast lists, character names or distinctive lines in the subtitles may still allow models to identify a film and retrieve pretrained knowledge. The title/year-only probe and character-name replacement experiment in Appendix B.6 already demonstrate this risk, but they cover only limited subsets. The paper does not stratify the full results by release year, film familiarity, or model knowledge cutoff, so the overall effect of memorization on the main results remains unknown.

  • The subtitle-only input limits the generalizability of the results to film-rating capability. The paper acknowledges that subtitles do not capture visual action, cinematography, music, gestures, and other non-speech information, while real-world age-suitability and national rating decisions also depend on evidence such as graphic violence, nudity, horror imagery, and physical actions. The current results are therefore best interpreted as prediction from subtitle-visible evidence and should not be directly generalized to rating complete audiovisual film content.

  • The English-centered setup, bundled prompting, and structured-output recovery may introduce additional sources of variation. All six languages are used as inputs, but narrative outputs must be written in English and are compared with English references, so the results also reflect cross-lingual translation and English-generation ability. The language-safety evaluation is likewise limited to English subtitles. In addition, the narrative and cultural tasks place multiple fields in a single JSON response, which may create dependencies across fields or countries; this has not been tested against single-task or single-country prompting. Appendix B.5 reports initial structural-failure rates of 7.0%, 7.5%, and 8.5% for Qwen3 235B on Arabic, Persian, and Vietnamese narrative tasks, respectively. Although structural recovery reaches 100%, the paper should clarify whether repair and retry affect semantics and what additional cost they incur.

  • The disclosure of generative-AI use in Figures 1 and 2 may be incomplete. The main infographic content in both figures appears to extend beyond the statement that AI assistance was limited to generating decorative icons.

Questions for authors
  1. Can the authors report human performance on the narrative, age-suitability, and country-rating tasks under the same subtitle inputs, task definitions, output constraints, and scoring setup? Please also report the number of annotators, independent agreement, disagreement resolution, and adjudication for subtitle cleaning, label curation, manual audits, and judge meta-evaluation, clearly distinguishing a human baseline from agreement between LLM judges.

  2. Given that Appendix B.6 shows effects from title/year-only input and character-name replacement, can the authors stratify the results by release year, film familiarity, and model knowledge cutoff, and compare title/year-only with subtitle-only conditions? Would the authors consider releasing a larger entity-anonymized variant to further quantify the effect of film-identity memorization on the main results?

  3. For the cultural-rating tasks, can the authors compare bundled prompting with independent single-task and single-country prompting, and report changes in accuracy, over-strictness bias, and cross-country error correlations? Could Kids-in-Mind or other external metadata also be used for subset analyses to clarify which rating judgments are solvable from subtitle-visible evidence and which mainly depend on information outside the subtitle modality?

  4. Persian performs poorly on several tasks. Can the authors reanalyze this result while controlling for token count, truncation rate, subtitle-translation quality, tokenization, and matched-film conditions? For Qwen3 235B structured-output recovery, please report the repair/retry procedure, its cost, and whether any semantic changes occur.

  5. Please expand the AI Use Statement for Figures 1 and 2 to clarify whether the main infographic content used generative AI, which models or tools were used, and the extent of human modification.

Reviewer_4aYCOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose a benchmark for long-context film understanding from subtitles for LLMs.

The benchmark contains 1012 movies with six subtitle tracks in different languages, and seven tasks for evaluating LLM narrative understanding and cultural prediction; three of the narrative understanding tasks are scored with an LLM-as-a-judge approach.

The authors run the benchmark on nine LLMs and organize the results around four research questions: whether aggregate ranking reflects uniform capabilities across tasks, how robust movie understanding from subtitles is across languages, where cultural calibration fails, and whether judgments about language safety can be grounded in subtitle evidence.

Critical strengths

None

Critical weaknesses

i. Validity of the evaluation, especially for narrative understanding and generation. The three narrative tasks are scored exclusively by an LLM-as-a-judge without human validation. The judge belongs to the same family as two of the evaluated models, and the only validation provided is agreement with a second model. Moreover, the references are public IMDb and Kids-in-Mind text, plausibly memorized by the evaluated models, so reference-based judging cannot distinguish subtitle understanding from recall of the reference itself.

ii. Validity of the “long-context evaluation” claim. An average input length of 40–45K tokens and the absence of an analysis of performance by input length leave the “long-context” framing weakly supported, and the probes in Appendix B.6 raise a more fundamental concern. With title and year alone (Table 19), three of the five models score higher on plot than with the full subtitle track, and all five models score higher on genre micro-F1 (Table 21). When character names are swapped consistently in subtitles and references (Table 20), every model drops, and the strongest models drop most (Sol −0.48, Gemini −0.35), the opposite of what the alternative explanations the appendix lists (tokenization, name familiarity) would predict. The simplest reading is that performance reflects recognition of the film and retrieval of memorized plot knowledge rather than integration of subtitle evidence.

iii. In the released 20-film sample, the English subtitle track of id 563 (Logan) is an audio commentary rather than the film's dialogue. Such a track would pass every automated check in Table 5, so only a manual audit could have caught it. This raises serious concerns about the verification and audit described in the paper and about the quality of the dataset itself.

Questions for authors

i. How do the authors reconcile the claim that the benchmark measures long-context understanding with the results in Appendix B.6? Could they report the per-film correlation between title-only and subtitle-only scores, and an ablation that recognition cannot solve, e.g., truncated subtitle tracks or a subset of films released after the evaluated models’ training cutoff?

ii. How many of the 1012 films are actually evaluated for each task, model, and subtitle language? This is never stated explicitly.

iii. Which tasks are evaluated in which languages and for which models? Table 15 reports “where available” and “for completed English lanes”, suggesting incomplete coverage that is not explained.

Reviewer_huKkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

CineSubBench evaluates LLMs on film understanding from subtitles alone. It contains 1,012 films (1977–2026), each with complete subtitle tracks in six languages (6,072 tracks, 8.13M entries) and ratings from ten national classification systems, assembled from Kids-in-Mind, Common Sense Media, IMDb and OpenSubtitles/SubDL and selected by a complete-coverage optimisation. Seven tasks run on ~40–45k-token inputs: plot, synopsis and key-message generation (scored 1–5 by GPT-5.6 Luna against IMDb/Kids-in-Mind references), multi-label genre, Common Sense age suitability, country-specific ratings within each national label space, and English-only language-safety evidence extraction against regex-derived gold. Nine LLMs are compared at temperature 0 under a declared composite. Gemini 3.8 Flash leads, ahead of GPT-5.6 Sol and DeepSeek V4 Pro (paired bootstrap CIs exclude zero); premises are recovered better than full synopses; scores drop from English to other subtitle languages, Persian lowest; national systems show distinct calibration; strong profanity is grounded far better than mild obscenity. Appendices add a second-LLM judge audit, title/year-only and name-swap probes, a Ministral scaling study and extensive per-language tables. Code and a 20-film sample are released; the full data sit on Google Drive.

Critical strengths

Critical Strengths

  1. A new, well-engineered resource. No prior benchmark gives text-only LLMs film-length subtitle tracks in six languages with matched ratings from ten national systems over the same films. The construction pipeline is documented stage by stage, provenance is per field, sanitisation of distribution boilerplate works (I checked the released English tracks), and the evaluation toolkit is clean: its tests pass and its metric code matches Equations 2–9. This matters because the dataset keeps its value even if the narrative claims are re-framed.

  2. The cultural and cross-lingual analyses are statistically honest. Paired film-level bootstrap and randomisation tests with Holm correction for the composite; cross-lingual CIs that correctly show which small deltas are not separated from zero; national label spaces kept distinct with majority-baseline lift; error-direction analysis. These parts are properly identified and are the benchmark's strongest use case.

Critical weaknesses
  1. The headline claim — long-form narrative understanding — is contradicted for the models that matter by the paper's own probe. Appendix B.6, Table 19: given only the title and release year, with no subtitles, GPT-5.6 Sol scores +0.07 higher on the narrative tasks than with the full subtitle track; Gemini −0.02; DeepSeek −0.04 (CIs include zero). For genre, title/year alone beats subtitle input for all five models tested (+2.5 to +11 pp, Table 21). The films are mainstream, Kids-in-Mind-reviewed releases and the references are IMDb text; frontier models recognise the film from dialogue and recall the metadata. In the released sample 9 of 20 films carry their title verbatim in a subtitle track, and all 20 are widely known. The name-swap drop (Sol −0.48, Table 20) fits recognition being disrupted. The appendix reports all this candidly; the abstract, Contribution 1 and §6 draw the opposite conclusion. Contemporaneous work on exactly this problem treats title/year parity as contamination and filters until it disappears (Bamman et al., arXiv:2608.21430). As framed, the narrative leaderboard measures recall of IMDb text for strong models and comprehension only for weak ones. Fixing this requires re-filtering and re-evaluation.

  2. The narrative evaluation has no human anchor and is biased against the outputs by construction. The only judge is an LLM scoring against IMDb references; the only validation is a second LLM, reported as pooled agreement (exact 56.8%; "98.4% within one point" on a five-point scale is a low bar) with a −0.19 offset, while the headline Sol–Gemini gaps are 0.02–0.06. The judge never sees the subtitles, so "invented event" means "not in the IMDb synopsis", not "not in the film". Candidates are capped at 300 words against references averaging 1,448 words (Table 10), so "79.7% of synopses omit a major event" is largely guaranteed. The judge and the second-ranked model are the same vendor family and no per-model agreement is reported. And the released sample contains a film (Logan) whose English "subtitle" track is a director's audio commentary; its language-safety gold was derived from that text (17 F-words vs. 53 in the Kids-in-Mind reference the authors say they audited against), so both the verification layers and the audit missed it. Nothing further can be checked because no outputs are released.

Questions for authors
  1. Run the title/year-only probe on all 1,012 films for all nine models and report narrative and genre results on the subset where title/year-only performance is near the subtitle-free floor. Do the model ranking and the Persian/cross-lingual findings hold there? If yes, Weakness 1 is resolved and I move to acceptance.
  2. Can you add a human rating study (even 100–200 synopses) with agreement against the judge, report Luna–Sonnet agreement per evaluated model, and run a judge variant that receives the subtitles?
  3. Why cap synopses at 300 words against 1,448-word references? Please separate omission from invention, or score against length-matched references.
  4. How many of the 6,072 tracks are commentary, trailer or other non-dialogue tracks? Please report the detection method and the affected count, and re-derive the language-safety gold for affected films.
  5. Will you release the saved model and judge outputs, the metric files behind every table, the OpenRouter/local inference code, and the construction code? The reproducibility statement says all of this is available; the repository does not contain it.

Suggestions, not part of my decision: rename the claim to "film understanding from subtitles" unless Q1 succeeds; consider a recognition-robust split of older or lesser-known films, as MF² does; reconcile the strict LC F1 definition (micro in Eqs. 6–7, macro in code); engage with self-preference results on LLM judges (arXiv:2404.13076, 2410.21819) and with FABLES' finding that LLM faithfulness raters correlate weakly with humans.

Minor, not part of my decision: §5.2 "decreases for all nine models" needs the App. D.4 caveat (two CIs include zero); the language-safety gold is a wordlist match (a dozen regexes recover 100% of strong-profanity gold on the sample), so call it lexical-evidence extraction rather than "safety judgments"; the LC task runs in 500-line chunks, so it is not long-context; pin requirements.txt.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.