- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper introduces LF-VideoQA, an open-ended question-answering dataset for Theory-of-Mind understanding in long videos. Using screenplays as the source of narrative structure, the authors construct the benchmark through scene parsing, subtitle time alignment, character belief and meta-belief extraction, LLM-based question generation, claim verification, and question-specific rubrics. They evaluate a range of vision-language models, agents, and a screenplay oracle. The central claim of the paper is that long-video understanding cannot be reduced to local clip retrieval. Models must instead integrate narrative evidence across minute-level or even hour-level time spans.
Critical strengths
The dataset emphasizes long-range temporal dependencies, character mental states, and open-ended long-form answers. It can therefore evaluate whether models understand belief changes and cross-scene causal relationships, rather than merely locating individual frames or short clips. The paper compares screenplay, subtitle, video-caption, direct-video, retrieval-agent, and oracle-agent settings, and analyzes recall, precision, contradiction rates, and performance across question types. This provides a useful diagnosis of system bottlenecks.
Critical weaknesses
-
Scene segmentation, subtitle alignment, character coreference, belief and meta-belief extraction, and claim verification may all introduce cascading errors. The current human study primarily evaluates the final questions, answers, and claims, which is insufficient to identify the source of errors.
-
The authors manually inspect only 150 out of 6,327 questions, approximately 2.4%. Section 3.5 reports two annotators and exact agreement ranging from 80.6% to 96.6%, which is encouraging, but chance-corrected agreement such as Cohen's kappa or Krippendorff's alpha is still missing. In addition, the paper validates agreement among different LLM judges but does not evaluate agreement between the judge and human annotators on model answers.
Questions for authors
-
Were the 150 manually inspected questions and their associated claims judged against the screenplay or the final released film?
-
What are the extraction accuracies for scene segmentation, subtitle alignment, character coreference, belief extraction, and claim verification? Can the authors provide at least one stage-wise error-propagation analysis?
-
Appendix G compares three LLM judges. Can the authors add a smaller human-annotated subset and report human-vs-LLM judge agreement and the main types of disagreement?
3 comments
Response to Reviewer JESp – W2
We thank the reviewer for highlighting the need for additional details regarding our human evaluation and inter-annotator agreement.
To clarify, our human study involved five annotators evaluating 150 questions sampled from 15 films, with each question independently evaluated by at least two annotators. Annotators were provided with the full screenplay and corresponding film and answered 12 multiple-choice questions assessing three aspects of dataset quality:
- Question-quality (clarity 1–3, answerability 1–3, sufficiency 1–3, verdict 1–2)
- Answer-quality (correctness 1–4, relevance 1–3, completeness 1–4, verdict 1–2)
- Claim-quality (understandability 1–3, support 1–4, relevance 1–3, evidence clip 1–5)
Across the study, the average pairwise Cohen's κ was 0.47, indicating moderate agreement. We additionally examined the effect of annotation time, as one annotator completed their assignments substantially faster than the others. In a sensitivity analysis excluding this annotator, average pairwise Cohen's κ increased to 0.66, indicating substantial agreement. We report both results for transparency, retaining the full-annotator results as our primary analysis.
We also provide a more detailed breakdown of the human evaluation results below:
| Question (keep) | Answer (keep) | Question AND Answer (keep) | |
|---|---|---|---|
| micro: all annotations | .91 | .92 | .87 |
| macro: average across annotators | .88 | .90 | .83 |
| all raters agreed | .83 | .85 | .77 |
The previously reported 76.7% corresponds to the strictest criterion, requiring unanimous approval of both the question and answer among the assigned annotators. This should not be interpreted as indicating that the remaining 23.3% of samples are necessarily invalid. The micro- and macro-aggregated results provide complementary perspectives, showing consistently high approval rates across the evaluated questions and answers.
We appreciate the reviewer's suggestion and will revise Section 3.5 to clarify the annotation protocol, report chance-corrected agreement statistics, and include a more detailed breakdown of the human study in the appendix.
Response to Reviewer JESp – Q1 + Q3
Q1: Were the 150 manually inspected questions and their associated claims judged against the screenplay or the final released film?
The human annotators were provided with access to both the full screenplay and the corresponding released film when evaluating the questions, answers, and associated claims.
Q3: Human–LLM judge agreement and disagreement analysis.
We thank the reviewer for this suggestion. Our original LLM-judge agreement experiment (Appendix G) was designed to assess whether rubric-based evaluation results were consistent across different LLM judges. To this end, we sampled 200 questions and compared the judgments of three models (gpt-5.6-luna, gpt-5.6-terra, and gemini-3.8-flash).
We agree that cross-model agreement alone does not fully establish alignment with human judgments. To further validate our evaluation framework, we conducted an additional human study involving two annotators who independently evaluated the same 200 questions. The annotators followed the same evaluation instructions provided to the LLM judges (Table 26, Appendix G), labeling each rubric claim as asserted, contradicted, or omitted.
We report the resulting human–LLM and inter-annotator agreement below:
| Comparison | Exact agreement | Cohen's κ |
|---|---|---|
| Human vs gpt-5.6-luna (primary judge) | 89.1% | 0.77 |
| Human vs gpt-5.6-terra | 88.3% | 0.76 |
| Human vs gemini-3.8-flash | 86.2% | 0.67 |
| Human 1 vs Human 2 | 78.3% | 0.54 |
| gpt-5.6-luna vs gpt-5.6-terra | 93.1% | 0.78 |
| gpt-5.6-luna vs gemini-3.8-flash | 92.8% | 0.73 |
| gpt-5.6-terra vs gemini-3.8-flash | 92.0% | 0.71 |
The results show substantial agreement between human judgments and all three LLM judges (Cohen's κ = 0.67–0.77), providing additional evidence supporting the reliability of our rubric-based evaluation. We will incorporate these findings and further details of the human evaluation protocol into Appendix G.
Response to Reviewer JESp – W1 + Q2
We Thank the reviewer for their comments, we address each of the weaknesses in this section
W1+Q2: We understand that there is no direct evaluation for every stage of the pipeline; however, we do not believe that it is necessary as the only time any of this processing is used is the creation of the (question, answer, rubric). Nothing is used on the main long-video question-answering task. Therefore, it is only important to ensure that the question, answer and rubric used in the dataset is factually correct, and that the claims used for the generation of that question (i.e., events, first-order beliefs, meta beliefs) are all factually correct too.
The pipeline is designed as “propose, then verify”, so an error in an early stage does not carry through to the released data. In Section 3.2 we explain "Because the belief record is extracted automatically, it can be wrong, and so can the claims drafted from it. We therefore verify every claim against the screenplay itself". We further explain how this verification process is done by outlining the 3 stages used to filter out the dataset in Appendix F. Whenever we validate the factuality and correctness of a claim, we verify it with GPT-Luna against the full film-script.
In this table we show the percentage of claims discarded before the generation of the questions. Once a claim is discarded, no question, answer or rubric ever uses it.