acceptodds
Under review as a conference paper at ICLR 2027

LF-VideoQA: Evaluating Long-Video Understanding with Long-Form Theory-of-Mind Questions

Abstract

The hardest part of understanding a long video is not pinpointing any one moment but synthesizing many, e.g., answering a question that requires tracking what develops across hours of content and articulating how the pieces fit together. To date, most long-video QA datasets require only a short answer or a multiple-choice selection, oversimplifying the long-video understanding task to retrieval. We introduce LF-VideoQA, a dataset for long-form, open-ended question answering over long narrative videos built on theory-of-mind questions: a character's belief depends on everything they have seen or been told, so each question demands assembling and reasoning from evidence distributed across the video. The dataset contains more than 6,300 questions on 94 recent films; each requires reasoning over scenes spanning an average of 23 minutes and comes with a question-specific rubric for recall and precision scoring. The dataset is created entirely automatically by annotating screenplays with scene events and each character's first-order beliefs and meta-beliefs, then generating (question, answer, rubric) triples through LLM-based question generation and verification; human studies validate their quality. We evaluate four families of state-of-the-art open-source VLMs across three settings and find the task highly challenging: (i) even an oracle with access to the screenplay falls well short of complete answers; (ii) the best non-oracle model attains barely half of the recall level of the oracle; (iii) unlike on prior long-video benchmarks, captions and subtitles underperform the video itself, indicating that the questions need visual evidence; and (iv) recall is the predominant bottleneck, with answers omitting many required rubric claims.

Primary area:
datasets and benchmarks
Keywords:
Long-Form Question Answering, LFQA, Video Question Answering, Long Video Question Answering, Video QA Dataset, Open-Ended Video Question Answering

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_JESpOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces LF-VideoQA, an open-ended question-answering dataset for Theory-of-Mind understanding in long videos. Using screenplays as the source of narrative structure, the authors construct the benchmark through scene parsing, subtitle time alignment, character belief and meta-belief extraction, LLM-based question generation, claim verification, and question-specific rubrics. They evaluate a range of vision-language models, agents, and a screenplay oracle. The central claim of the paper is that long-video understanding cannot be reduced to local clip retrieval. Models must instead integrate narrative evidence across minute-level or even hour-level time spans.

Critical strengths

The dataset emphasizes long-range temporal dependencies, character mental states, and open-ended long-form answers. It can therefore evaluate whether models understand belief changes and cross-scene causal relationships, rather than merely locating individual frames or short clips. The paper compares screenplay, subtitle, video-caption, direct-video, retrieval-agent, and oracle-agent settings, and analyzes recall, precision, contradiction rates, and performance across question types. This provides a useful diagnosis of system bottlenecks.

Critical weaknesses
  • Scene segmentation, subtitle alignment, character coreference, belief and meta-belief extraction, and claim verification may all introduce cascading errors. The current human study primarily evaluates the final questions, answers, and claims, which is insufficient to identify the source of errors.

  • The authors manually inspect only 150 out of 6,327 questions, approximately 2.4%. Section 3.5 reports two annotators and exact agreement ranging from 80.6% to 96.6%, which is encouraging, but chance-corrected agreement such as Cohen's kappa or Krippendorff's alpha is still missing. In addition, the paper validates agreement among different LLM judges but does not evaluate agreement between the judge and human annotators on model answers.

Questions for authors
  • Were the 150 manually inspected questions and their associated claims judged against the screenplay or the final released film?

  • What are the extraction accuracies for scene segmentation, subtitle alignment, character coreference, belief extraction, and claim verification? Can the authors provide at least one stage-wise error-propagation analysis?

  • Appendix G compares three LLM judges. Can the authors add a smaller human-annotated subset and report human-vs-LLM judge agreement and the main types of disagreement?

3 comments
AuthorsOpenReview ↗

Response to Reviewer JESp – W1 + Q2

We Thank the reviewer for their comments, we address each of the weaknesses in this section

W1+Q2: We understand that there is no direct evaluation for every stage of the pipeline; however, we do not believe that it is necessary as the only time any of this processing is used is the creation of the (question, answer, rubric). Nothing is used on the main long-video question-answering task. Therefore, it is only important to ensure that the question, answer and rubric used in the dataset is factually correct, and that the claims used for the generation of that question (i.e., events, first-order beliefs, meta beliefs) are all factually correct too.

The pipeline is designed as “propose, then verify”, so an error in an early stage does not carry through to the released data. In Section 3.2 we explain "Because the belief record is extracted automatically, it can be wrong, and so can the claims drafted from it. We therefore verify every claim against the screenplay itself". We further explain how this verification process is done by outlining the 3 stages used to filter out the dataset in Appendix F. Whenever we validate the factuality and correctness of a claim, we verify it with GPT-Luna against the full film-script.

In this table we show the percentage of claims discarded before the generation of the questions. Once a claim is discarded, no question, answer or rubric ever uses it.

Claim rests onDiscarded by the screenplay check
Scene events only6.6%
Meta-belief24.1%
First-order belief revision28.7%
First-order false belief32.9%
AuthorsOpenReview ↗

Response to Reviewer JESp – W2

We thank the reviewer for highlighting the need for additional details regarding our human evaluation and inter-annotator agreement.

To clarify, our human study involved five annotators evaluating 150 questions sampled from 15 films, with each question independently evaluated by at least two annotators. Annotators were provided with the full screenplay and corresponding film and answered 12 multiple-choice questions assessing three aspects of dataset quality:

  • Question-quality (clarity 1–3, answerability 1–3, sufficiency 1–3, verdict 1–2)
  • Answer-quality (correctness 1–4, relevance 1–3, completeness 1–4, verdict 1–2)
  • Claim-quality (understandability 1–3, support 1–4, relevance 1–3, evidence clip 1–5)

Across the study, the average pairwise Cohen's κ was 0.47, indicating moderate agreement. We additionally examined the effect of annotation time, as one annotator completed their assignments substantially faster than the others. In a sensitivity analysis excluding this annotator, average pairwise Cohen's κ increased to 0.66, indicating substantial agreement. We report both results for transparency, retaining the full-annotator results as our primary analysis.

We also provide a more detailed breakdown of the human evaluation results below:

Question (keep)Answer (keep)Question AND Answer (keep)
micro: all annotations.91.92.87
macro: average across annotators.88.90.83
all raters agreed.83.85.77

The previously reported 76.7% corresponds to the strictest criterion, requiring unanimous approval of both the question and answer among the assigned annotators. This should not be interpreted as indicating that the remaining 23.3% of samples are necessarily invalid. The micro- and macro-aggregated results provide complementary perspectives, showing consistently high approval rates across the evaluated questions and answers.

We appreciate the reviewer's suggestion and will revise Section 3.5 to clarify the annotation protocol, report chance-corrected agreement statistics, and include a more detailed breakdown of the human study in the appendix.

AuthorsOpenReview ↗

Response to Reviewer JESp – Q1 + Q3

Q1: Were the 150 manually inspected questions and their associated claims judged against the screenplay or the final released film?

The human annotators were provided with access to both the full screenplay and the corresponding released film when evaluating the questions, answers, and associated claims.

Q3: Human–LLM judge agreement and disagreement analysis.

We thank the reviewer for this suggestion. Our original LLM-judge agreement experiment (Appendix G) was designed to assess whether rubric-based evaluation results were consistent across different LLM judges. To this end, we sampled 200 questions and compared the judgments of three models (gpt-5.6-luna, gpt-5.6-terra, and gemini-3.8-flash).

We agree that cross-model agreement alone does not fully establish alignment with human judgments. To further validate our evaluation framework, we conducted an additional human study involving two annotators who independently evaluated the same 200 questions. The annotators followed the same evaluation instructions provided to the LLM judges (Table 26, Appendix G), labeling each rubric claim as asserted, contradicted, or omitted.

We report the resulting human–LLM and inter-annotator agreement below:

ComparisonExact agreementCohen's κ
Human vs gpt-5.6-luna (primary judge)89.1%0.77
Human vs gpt-5.6-terra88.3%0.76
Human vs gemini-3.8-flash86.2%0.67
Human 1 vs Human 278.3%0.54
gpt-5.6-luna vs gpt-5.6-terra93.1%0.78
gpt-5.6-luna vs gemini-3.8-flash92.8%0.73
gpt-5.6-terra vs gemini-3.8-flash92.0%0.71

The results show substantial agreement between human judgments and all three LLM judges (Cohen's κ = 0.67–0.77), providing additional evidence supporting the reliability of our rubric-based evaluation. We will incorporate these findings and further details of the human evaluation protocol into Appendix G.

Reviewer_y5mDOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper targets the gap in video QA benchmarks between short-answer questions and the long-input, long-output reasoning that feature-length films demand. It proposes LF-VideoQA, a benchmark of open-ended theory-of-mind questions over recent films, where each question probes a character's belief or meta-belief at an anchored moment, requires evidence distributed across multiple scenes, and is scored by an LLM judge against a claim-level rubric. The entire construction pipeline screenplay parsing, belief-record extraction, question/answer/rubric generation and verification is automatic, with small human and multi-judge studies as quality checks.

Critical strengths

1. Automatic pipeline with disciplined anti-leakage and answerability rules. Question-writing rules forbid anchors that reveal belief values or use names never spoken aloud, require concrete distractors and anchor-location rivals, and discard thin context-sets.

2. Dataset statistics defeat protagonist-tracking and highlight-retrieval shortcuts. Questions cover a median of 15.5 characters per film, cite a median of 71% of scenes with near-uniform anchor placement, and every question cites at least two scenes.

3. Claim-level rubric metric with self-consistency safeguards. The verifier defaults to "unsupported" when unsure, and every gold answer is re-graded against its own rubric with questions retired if the gold omits a core claim.

Critical weaknesses

1. No question-only or closed-book contamination control, despite the paper's own motivating critique. The introduction faults prior benchmarks because "even the question alone can support strong performance" (p.2), yet Table 3 contains no blind baseline, and the sole mitigation, i.e., films released 2023+ (Section 3.1, p.4), does not protect against 2025–2026 training data covering widely discussed films whose screenplays are themselves public. Without this control, the design cannot separate memorized plot knowledge from the distributed visual reasoning the benchmark claims to require.

2. Ground truth is the screenplay, not the film models actually watch. Rubric claims are verified only against screenplay text ("Judge ONLY from the text shown," Table 24, p.49), only anchors, not cited evidence scenes, require subtitle alignment (Appendix C, p.15), and no audit quantifies screenplay–film divergence. This construct-validity gap confounds the central oracle-vs-video comparison, since some rubric claims may be supported only by text a viewer cannot see.

3. Circular construction and scoring concentrated in one model family, with the judge never validated against humans. gpt-5.6-sol both writes and verifies questions, and same-family gpt-5.6-luna is the primary judge; Appendix G measures only agreement among three LLM judges sharing an identical prompt, so prompt-induced biases common to all judges are structurally undetectable, and the human annotators never graded judge verdicts.

W4. Precision metric does not operationalize the claimed faithfulness construct. P = |A|/(|A|+|X|) (Eq. 1) covers only rubric claims and the judge is instructed that extra answer content "must not influence any verdict" (Table 25), so hallucinated content outside the rubric goes unpenalized, yet the conclusion claims rubrics measure "faithfulness".

Questions for authors

The authors are expected to stress the questions stressed in the weakness section.

5 comments
AuthorsOpenReview ↗

Response to Reviewer y5mD – W1

We Thank the reviewer for their comments, we address each of the weaknesses in this section.

The reviewer is right, depending on only the release date is not sufficient to claim protection against prior knowledge. We run two additional experiments to show that the results are immune to the prior knowledge:

  1. Title-only: the model answers each question knowing only the film's title. This is the question-only baseline the reviewer asks for.
  2. Title → summary: the model first writes a summary of the film from its title alone, and then answers from that summary. This is the film-identity probe the reviewer asks for, scored on whether what the model recalls about the film is enough to answer our questions.
ModelTitle-only F1Title → summary F1Subtitles F1Screenplay F1
Qwen3.6-35B.029.004.165.505
Qwen3.8-27B.022.001.175.544
Gemma4-26B.046.007.064.290

The F1 scores are significantly less than the Screenplay and Subtitle variants. Therefore, proving that indeed from the Title alone, you cannot answer the questions. We will add both settings to paper in the camera ready version.

AuthorsOpenReview ↗

Response to Reviewer y5mD – W2

We Thank the reviewer for their comments, we address each of the weaknesses in this section.

W2: We use screenplays as our primary reference because they are explicitly written to guide the production of the film itself. Unlike plot summaries or other secondary textual descriptions, screenplays contain detailed scene descriptions, character actions, interactions, and dialogue that directly correspond to the narrative presented on screen. Consequently, we expect a high degree of correspondence between screenplay content and the final film. While video-based verification using VLMs is an alternative, it introduces additional uncertainty from the visual reasoning capabilities of the evaluator itself. In addition, most LLM-as-a-judge benchmarks and datasets for open-ended video question answering use text-based evaluation for the same exact reason stated. Text-based evaluation allows us to verify narrative claims against an explicit reference without introducing another potentially unreliable video-understanding component. This is particularly important for complex character beliefs and meta-beliefs that require integrating information across multiple scenes.

AuthorsOpenReview ↗

Response to Reviewer y5mD – W3

We thank the reviewer for raising this concern. To mitigate potential model-specific biases, we evaluated judge agreement across three different models, including models from different families (gpt-5.6-luna, gpt-5.6-terra, and gemini-3.8-flash). Additionally, the model used for dataset generation is distinct from the primary evaluation judge.

We agree that cross-model agreement alone may not fully capture shared evaluation biases. To further validate the reliability of our evaluation, we conducted an additional human study in which two annotators independently evaluated the same 200 samples. We report the agreement between human judgments and each LLM judge, as well as inter-human and inter-model agreement.

ComparisonExact agreementCohen's κ
Human vs gpt-5.6-luna (primary judge)89.1%0.77
Human vs gpt-5.6-terra88.3%0.76
Human vs gemini-3.8-flash86.2%0.67
Human 1 vs Human 278.3%0.54
gpt-5.6-luna vs gpt-5.6-terra93.1%0.78
gpt-5.6-luna vs gemini-3.8-flash92.8%0.73
gpt-5.6-terra vs gemini-3.8-flash92.0%0.71

The results demonstrate substantial agreement between human judgments and all three LLM judges (Cohen's κ = 0.67–0.77), providing additional evidence supporting the reliability of our evaluation framework. We will incorporate these results into Appendix G.

Reviewer_fSWDOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

Most long-video QA datasets require simple perception and retrieval of what is happening at one moment in the video, which is a somewhat oversimplified task. The authors propose a new task and dataset called LF-VideoQA, built on theory-of-mind questions, i.e. understanding what each character believes at a specific time. A character’s belief depends on everything they have seen or been told, as a result this task requires assembling and reasoning from evidence distributed across the video. The dataset contains more than 6,300 questions on 94 recent films (all movies from 2023 or later).

The dataset construction pipeline primarily depends on screenplays which are treated as oracles. The screenplay is pre-process to extract scenes with speakers and dialogue, then the scenes are split into small events tagged with who took part and who witnessed them. Qwen3.5-27B is used for both these steps. After this information is extracted gpt-5.6-sol is used to write the questions. For one character, one anchor scene and one question type (11 types), the writer gets a small package with the relevant belief entries, nearby scenes and the allowed names. There are verification passes to drop invalid questions and check every claim against the actual screenplay text.

At test time, gpt-5.6-luna as judge marks each rubric claim in a model's answer as asserted, contradicted or omitted. Recall, precision and F1 are computed from those labels.

All tested models struggle: even with the full screenplay (oracle setting) the best model gets only about half the required claims right (F1 0.54), without the screenplay the best video-based setup (video plus subtitles) achieves an F1 score of 0.30.

Critical strengths
  • The proposed task of theory of mind questions for video QA evaluation is interesting and novel, and represents a major jump in video understanding tasks. What a character believes at a given moment depends on everything they have witnessed, been told, or failed to see up to that point. Questions built this way naturally need evidence from several scenes spread across the film.
Critical weaknesses
  • As noted in the human study on the small 150 question sample set, "both components received unanimous keep decisions for 76.7% of questions", i.e. >23% of the dataset might not be of acceptable quality. Given this issue, it might make sense to do a second pass over the dataset, using LLM/VLM as judge + some human intervention to filter out such bad samples from the broader dataset.

  • gpt-5.6-luna is used as the main judge for scoring, agreement of this judge is evaluated with gpt-5.6-terra and gemini-3.8-flash on a 200 question subset. However, agreement with humans is not measured.

  • The only protection against prior knowledge is "films released in 2023 or later". The models tested are 2026 releases, and these films (Deadpool & Wolverine, Conclave, Nosferatu, The Wild Robot) have Wikipedia plot summaries, fan wikis and reviews.

Questions for authors
  • Is a question-only baseline tested to filter out trivial samples ?
  • Report a "film identity" probe asking each model to summarise the plot from the title alone to check for which titles are already memorized by the models.
  • Measure judge-vs-human agreement on scoring
  • The refusal rate is almost 90% for several settings, can we report results with forced answer setting ?
4 comments
AuthorsOpenReview ↗

Response to Reviewer fSWD – W1

W1: Clarification on the human study conducted and inter-annotator agreement scores.

The study was conducted on a total of 5 annotators. We sampled 150 questions from 15 films. For each question, there has to be at least 2 annotators (with most being exactly 2). The average time spent by each annotator is about 4.6 hrs. Given a question, the full film-script and the full film-video, the annotators would answer 12 multiple-choice questions.

  • Question-quality (clarity 1–3, answerability 1–3, sufficiency 1–3, verdict 1–2)
  • Answer-quality (correctness 1–4, relevance 1–3, completeness 1–4, verdict 1–2)
  • Claim-quality (understandability 1–3, support 1–4, relevance 1–3, evidence clip 1–5)

The average pairwise Cohen's κ score is 0.47, which is considered to be moderate agreement. Upon further investigation, one of the annotators only spent 1 hour, significantly less than the others (8.9, 8.1, 3.2, 2 hours respectively); without their annotation the average pairwise Cohen's κ score is higher, 0.66.

Here is a detailed summary of each annotator result across the 150 question study. The strictest measure shows that the percentage of question-answers that ALL annotators agreed on is 0.767; however, as established earlier, not all annotators spent the same amount of time. Therefore, we believe that using the micro or macro measure is a more realistic measure of the quality of the dataset. These measures show that indeed the dataset is sufficiently of good quality.

Question (keep)Answer (keep)Question AND Answer (keep)
micro: all annotations.91.92.87
macro: average across annotators.88.90.83
all raters agreed.83.85.77

We thank the reviewer for raising this issue and we will be making these numbers present and clarified in Section 3.5. We will also give a much more detailed explanation of the human study in the Appendix.

AuthorsOpenReview ↗

Response to Reviewer fSWD – W2+Q3

W2+Q3: Clarification on the LLM-Judge experiment and a new conducted human study to evaluate human preference alignment with the LLM-judge results.

In the paper, we wanted to prove that the metric/rubric used for each question is not dependent on the LLM model used as a judge.​​ Therefore, we sampled 200 questions and evaluated the agreement scores across three models. We have not done a human study in which the annotator acts as a judge.

We, therefore, run this additional study where we hired two annotators to annotate the same 200 questions. We provided the human annotators with the same prompt the LLM judge is provided with (Table 26, Appendix G): for each claim in the rubric, label it as asserted, contradicted or omitted.

ComparisonExact agreementCohen's κ
Human vs gpt-5.6-luna (primary judge)89.1%0.77
Human vs gpt-5.6-terra88.3%0.76
Human vs gemini-3.8-flash86.2%0.67
Human 1 vs Human 278.3%0.54
gpt-5.6-luna vs gpt-5.6-terra93.1%0.78
gpt-5.6-luna vs gemini-3.8-flash92.8%0.73
gpt-5.6-terra vs gemini-3.8-flash92.0%0.71

Cohen's κ scores between the human judge and the LLM judges fall in the "substantial agreement" range. We will add these results to Appendix G.

AuthorsOpenReview ↗

Response to Reviewer fSWD – W3+Q1+Q2

W3+Q1+Q2: The reviewer is right, depending on only the release date is not sufficient to claim protection against prior knowledge. Following the reviewers suggestion we run two additional experiments:

  1. Title-only: the model answers each question knowing only the film's title. This is the question-only baseline the reviewer asks for.
  2. Title → summary: the model first writes a summary of the film from its title alone, and then answers from that summary. This is the film-identity probe the reviewer asks for, scored on whether what the model recalls about the film is enough to answer our questions.
ModelTitle-only F1Title → summary F1Subtitles F1Screenplay F1
Qwen3.6-35B.029.004.165.505
Qwen3.8-27B.022.001.175.544
Gemma4-26B.046.007.064.290

The F1 scores are significantly less than the Screenplay and Subtitle variants. Therefore, proving that indeed from the Title alone, you cannot answer the questions. We will add both settings to paper in the camera ready version.

AuthorsOpenReview ↗

Response to Reviewer fSWD – Q4

Q4: The model still gives an answer in these cases. What we count as a refusal is an answer that opens with a hedge, something like "There is no direct evidence… However, …", and then goes on to answer the question. The judge scores the full answer, so an answer that starts with a refusal can still receive recall, precision and F1. The reported results are therefore already close to a forced-answer setting.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.