acceptodds
Withdrawn from ICLR 2027

DramaVQA: Probing Detailed Understanding of Ultra-Long TV Dramas

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Despite recent progress in long-form video understanding, current benchmarks mostly focus on short clips or hour-scale videos. This leaves a major gap in evaluating ultra-long video understanding, where videos often last tens of hours. To bridge this gap, we introduce DramaVQA, a large-scale benchmark built upon long-form TV dramas. Containing approximately 24K question-answering (QA) pairs, the scale of DramaVQA exceeds previous long-video datasets, such as Video-MME and LVBench, by an order of magnitude. Our benchmark has two main features that make it stand out from existing datasets. First, we establish a scalable, LLM-assisted annotation pipeline that uses multimodal large language models (MLLMs) to generate complex questions, followed by strict human checking to ensure high data quality. Second, DramaVQA introduces a multi-level time structure; queries are designed across different video lengths, with the hardest questions requiring overall understanding across the entire drama. Extensive evaluations show that state-of-the-art MLLMs face major challenges when tested on these very long videos, highlighting the need for agentic AI algorithms capable of managing long-term memory. We will release all data and code to facilitate further research.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Long video understanding, Benchmark, TV Drama

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bEHeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces DramaVQA for detailed understanding of long TV dramas. It contains 20K evaluation questions. The same questions are evaluated within progressively larger video contexts. The paper also studies caption-based memory and additional multi-point and multi-hop tasks.

Critical strengths
  • Evaluating the same questions across multiple context lengths provides a useful way to study the effect of context expansion.

  • The supplement documents human verification and input-condition ablations. The released annotations and evaluation code help readers inspect the benchmark.

Critical weaknesses
  • The main questions are generated and validated within short L2 segments. They are then evaluated within larger enclosing contexts. This primarily tests retrieval of local evidence under increasing distraction. It does not establish that answering requires reasoning across episodes. The separate multi-hop study is useful, but it does not resolve this concern for the main benchmark.

  • Sampling 256 or 512 frames from many hours can miss the relevant event entirely. Low accuracy may therefore reflect missing observations rather than failed reasoning. An oracle-evidence baseline is needed to separate these effects.

  • The subtitle protocol is inconsistent. Section 4.1 states that subtitles remain in the frames and must be read visually. Supplement Section C.2 states that direct subtitle text is adopted. Table 6 treats these as different conditions. The authors should specify which setting produced each main result.

  • The efficiency analysis includes caption construction costs, which is appropriate. However, its argument against shared computation for native MLLMs focuses on batching questions together. Reusing visual representations or a shared video prefix need not expose other questions or answers. The efficiency comparison should evaluate such reuse where supported.

  • The dataset versions are unclear. The released mini set contains 2,000 of the 20,000 questions. This agrees with Supplement Table 5 but not the main paper's 5% description. The main paper and supplement also report different multi-hop sample counts. These discrepancies should be reconciled.

Questions for authors
  • Which subtitle and speaker-information settings were used for the main results?

  • How do models perform when given the relevant evidence segments?

  • What fraction of the main benchmark requires information from multiple separated events?

  • Can you reconcile the subset sizes and multi-hop experiments across the paper, supplement, and release?

1 comment

Thank you again for your thoughtful comments. We have carefully addressed your concerns in our rebuttal and would greatly appreciate it if you could take a moment to review our responses.

We would be happy to provide further clarification or additional evidence if any of our responses remain unclear. Thank you for your time and consideration!

Reviewer_p6KpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DramaVQA contains approximately 20,000 questions over 12 TV dramas. The same question is evaluated under four enclosing contexts, from minutes to tens of hours. The construction uses character/subtitle metadata, hierarchical captions, language-prior filtering, and human checking. The main paper evaluates native multimodal models and retrieval/agent systems, reports agent post-training, and separately studies multi-point and multi-hop questions. This assessment covers the main paper only.

Critical strengths

Dense repeated querying of a coherent long recording is a meaningful workload for memory and retrieval systems. Approximately 1,600 questions per drama make shared representations relevant, rather than evaluating only a few questions per long video.

Holding the question fixed while expanding its enclosing context is a useful design for studying evidence localization. The construction also includes two-stage human checking and strong language-only filtering (Sec. 3.2).

The preliminary multi-point/multi-hop study is a positive addition. In particular, paired second-hop questions use the same input as their multi-hop counterparts, providing evidence of an additional composition challenge (Sec. 4.4; Table 3).

Critical weaknesses

The central task supports dense local-evidence retrieval more directly than long-range narrative reasoning. The main QA pairs originate from L2 captions averaging 1.2 minutes (p. 4), and are then placed in progressively longer contexts. Increasing context duration increases search difficulty, but does not establish that the answer requires integrating widely separated events. Under a fixed 256/512-frame budget, declining coverage of a short evidential event is also a plausible explanation for the accuracy drop; the paper itself notes this sampling effect (p. 7). Table 3 does not resolve the construct gap for the main 20K tasks: its separate composed questions use concatenated relevant L3 segments (p. 9). Retrieval is a legitimate contribution, but the evidence does not justify reading the main-track decline as a general failure of whole-drama reasoning. DramaQA [1] provides story/character-centered context; the new dense and much longer workload should be credited on its own precise terms, not treated as equivalent to distributed narrative integration.

The main-paper experiments do not yet establish a sufficiently strong advance around the narrower memory/retrieval contribution. LongVideo-R1 [2] already provides hierarchical navigation; the distinctive opportunity here is what dense querying reveals about reusable memory and accuracy/cost tradeoffs. Sec. 4.3 states a batch-amortization advantage, but the main paper does not provide the quantitative decomposition needed to assess that claim. In Table 2, post-training improves accuracy while also increasing inference time, and p. 8 reports more tool calls, so the improvement is not a fixed-cost policy comparison. The 35.1% Ultra-Long full-set accuracy of native Qwen3.6 versus 30.5% for post-tuned LongVideo-R1 is also not a matched-backbone comparison and supports neither blanket agent superiority nor the converse. Within the main-paper scope, the evidence is therefore stronger for a challenging workload than for a distinctive finding about memory or efficient reasoning. This is a limit of the evidence assessed here, not a claim that supporting analyses were never performed elsewhere.

References:

Questions for authors

From the question construction and analyses described in the main paper, which conclusions concern locating local evidence, and which establish integration across separated evidence? Please keep the core 20K workload distinct from the composed-question study.

How should the main-paper agent results be interpreted after separating extra tool use/inference time from improved decision quality? Which memory or efficiency conclusion is quantitatively supported within the main-paper results, rather than inferred solely from longer inputs or potential preprocessing reuse?

1 comment

Thank you again for your thoughtful comments. We have carefully addressed your concerns in our rebuttal and would greatly appreciate it if you could take a moment to review our responses.

We would be happy to provide further clarification or additional evidence if any of our responses remain unclear. Thank you for your time and consideration!

Reviewer_X7JsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DramaVQA introduces 20K TV-drama questions evaluated with four video context lengths, averaging 7.6 minutes to 38.9 hours. It provides hierarchical annotations, human-checked questions, and evaluations of native and retrieval-based models. Accuracy decreases substantially as the video context expands. The paper also explores composed multi-point and multi-hop questions.

Critical strengths
  1. The dataset provides detailed questions and character, subtitle, and caption annotations over unusually long videos, making it a potentially useful resource for video retrieval and memory research.
  2. Reusing the same questions across context lengths makes comparisons easier to interpret. Human verification against the source videos also helps correct errors introduced during automatic question generation.
Critical weaknesses
  1. The videos are long, but the main questions can be answered from a short clip. The main 20K questions are generated and checked within clips averaging 1.2 minutes (§3.2). Putting these questions into much longer videos mainly makes the relevant clip harder to find. It does not require combining information from distant parts of the video. LvBench’s Godfather example illustrates this distinction by combining clues from widely separated scenes (Zhang et al., 2025, Fig. 1).

  2. Models may answer incorrectly because the sampled frames do not contain the information needed. Sampling 512 frames from 38.9 hours gives roughly one frame every 4.6 minutes, so a short relevant event can be missed entirely. The experiments do not separate missing evidence, failure to find evidence already provided, and failure to understand it. The accuracy drop therefore cannot be directly attributed to weaker long-video understanding.

  3. There is no subtitle-only result to show how much the images help. The appendix compares conditions with and without subtitles, but does not report how many questions can be answered using subtitles alone. The input descriptions also disagree: §4.1 says models read subtitles from frames, whereas Appendix C.2 says subtitle text is supplied directly. The actual inputs used for evaluation need clarification.

  4. Questions may become ambiguous when evaluated over the full drama. Appendix B.5 checks at most 50 retrieved clip descriptions. Another scene matching the question but giving a different answer may be missed by retrieval, or its relevant details may be absent from the description. No audit reports how often this happens. An unambiguous answer within the original clip does not establish an unambiguous answer across the full drama.

Questions for authors

I have no other questions, please refer to the weaknesses.

1 comment

Thank you again for your thoughtful comments. We have carefully addressed your concerns in our rebuttal and would greatly appreciate it if you could take a moment to review our responses.

We would be happy to provide further clarification or additional evidence if any of our responses remain unclear. Thank you for your time and consideration!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.