- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces CineSubBench, a benchmark that uses multilingual film subtitles to evaluate large language models (LLMs) on long-context narrative understanding and cross-cultural judgment. The benchmark contains 1,012 films released between 1977 and 2026 and is organized around a multi-task, multilingual, and multicultural (MultiX) setting. The authors evaluate nine closed and open-weight models and use GPT-5.6 Luna as a reference-based, two-stage LLM judge. The results reveal factual errors in narrative generation and performance differences across languages in cultural and safety-related evaluation.
Critical strengths
The dataset has broad scale and multilingual coverage: it integrates multiple sources through IMDb identifiers and covers six subtitle languages—English, Arabic, Indonesian, Persian, Romanian, and Vietnamese—with 1,012 films, 6,072 subtitle tracks, and 8.13M timestamped subtitle entries.
Critical weaknesses
-
The benchmark lacks a human baseline and sufficient reporting of annotation agreement. The paper reports high agreement between independent LLM judges, but does not report human performance under the same subtitle inputs, task definitions, output constraints, and scoring setup. It is therefore unclear how solvable the tasks are, what a reasonable performance ceiling is, or how large the model–human gap may be. Low scores could reflect limited model capability, task difficulty, reference-label limitations, or scoring design, and these possibilities are difficult to distinguish. The paper should also report the number of annotators, independent annotation agreement, and disagreement-resolution procedures for subtitle cleaning, task-label curation, manual audits, and judge evaluation.
-
The impact of pretrained priors and film-identity memorization is not yet quantified. Although the standard evaluation hides film titles, release years, and cast lists, character names or distinctive lines in the subtitles may still allow models to identify a film and retrieve pretrained knowledge. The title/year-only probe and character-name replacement experiment in Appendix B.6 already demonstrate this risk, but they cover only limited subsets. The paper does not stratify the full results by release year, film familiarity, or model knowledge cutoff, so the overall effect of memorization on the main results remains unknown.
-
The subtitle-only input limits the generalizability of the results to film-rating capability. The paper acknowledges that subtitles do not capture visual action, cinematography, music, gestures, and other non-speech information, while real-world age-suitability and national rating decisions also depend on evidence such as graphic violence, nudity, horror imagery, and physical actions. The current results are therefore best interpreted as prediction from subtitle-visible evidence and should not be directly generalized to rating complete audiovisual film content.
-
The English-centered setup, bundled prompting, and structured-output recovery may introduce additional sources of variation. All six languages are used as inputs, but narrative outputs must be written in English and are compared with English references, so the results also reflect cross-lingual translation and English-generation ability. The language-safety evaluation is likewise limited to English subtitles. In addition, the narrative and cultural tasks place multiple fields in a single JSON response, which may create dependencies across fields or countries; this has not been tested against single-task or single-country prompting. Appendix B.5 reports initial structural-failure rates of 7.0%, 7.5%, and 8.5% for Qwen3 235B on Arabic, Persian, and Vietnamese narrative tasks, respectively. Although structural recovery reaches 100%, the paper should clarify whether repair and retry affect semantics and what additional cost they incur.
-
The disclosure of generative-AI use in Figures 1 and 2 may be incomplete. The main infographic content in both figures appears to extend beyond the statement that AI assistance was limited to generating decorative icons.
Questions for authors
-
Can the authors report human performance on the narrative, age-suitability, and country-rating tasks under the same subtitle inputs, task definitions, output constraints, and scoring setup? Please also report the number of annotators, independent agreement, disagreement resolution, and adjudication for subtitle cleaning, label curation, manual audits, and judge meta-evaluation, clearly distinguishing a human baseline from agreement between LLM judges.
-
Given that Appendix B.6 shows effects from title/year-only input and character-name replacement, can the authors stratify the results by release year, film familiarity, and model knowledge cutoff, and compare title/year-only with subtitle-only conditions? Would the authors consider releasing a larger entity-anonymized variant to further quantify the effect of film-identity memorization on the main results?
-
For the cultural-rating tasks, can the authors compare bundled prompting with independent single-task and single-country prompting, and report changes in accuracy, over-strictness bias, and cross-country error correlations? Could Kids-in-Mind or other external metadata also be used for subset analyses to clarify which rating judgments are solvable from subtitle-visible evidence and which mainly depend on information outside the subtitle modality?
-
Persian performs poorly on several tasks. Can the authors reanalyze this result while controlling for token count, truncation rate, subtitle-translation quality, tokenization, and matched-film conditions? For Qwen3 235B structured-output recovery, please report the repair/retry procedure, its cost, and whether any semantic changes occur.
-
Please expand the AI Use Statement for Figures 1 and 2 to clarify whether the main infographic content used generative AI, which models or tools were used, and the extent of human modification.