- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces DramaVQA for detailed understanding of long TV dramas. It contains 20K evaluation questions. The same questions are evaluated within progressively larger video contexts. The paper also studies caption-based memory and additional multi-point and multi-hop tasks.
Critical strengths
-
Evaluating the same questions across multiple context lengths provides a useful way to study the effect of context expansion.
-
The supplement documents human verification and input-condition ablations. The released annotations and evaluation code help readers inspect the benchmark.
Critical weaknesses
-
The main questions are generated and validated within short L2 segments. They are then evaluated within larger enclosing contexts. This primarily tests retrieval of local evidence under increasing distraction. It does not establish that answering requires reasoning across episodes. The separate multi-hop study is useful, but it does not resolve this concern for the main benchmark.
-
Sampling 256 or 512 frames from many hours can miss the relevant event entirely. Low accuracy may therefore reflect missing observations rather than failed reasoning. An oracle-evidence baseline is needed to separate these effects.
-
The subtitle protocol is inconsistent. Section 4.1 states that subtitles remain in the frames and must be read visually. Supplement Section C.2 states that direct subtitle text is adopted. Table 6 treats these as different conditions. The authors should specify which setting produced each main result.
-
The efficiency analysis includes caption construction costs, which is appropriate. However, its argument against shared computation for native MLLMs focuses on batching questions together. Reusing visual representations or a shared video prefix need not expose other questions or answers. The efficiency comparison should evaluate such reuse where supported.
-
The dataset versions are unclear. The released mini set contains 2,000 of the 20,000 questions. This agrees with Supplement Table 5 but not the main paper's 5% description. The main paper and supplement also report different multi-hop sample counts. These discrepancies should be reconciled.
Questions for authors
-
Which subtitle and speaker-information settings were used for the main results?
-
How do models perform when given the relevant evidence segments?
-
What fraction of the main benchmark requires information from multiple separated events?
-
Can you reconcile the subset sizes and multi-hop experiments across the paper, supplement, and release?
Thank you again for your thoughtful comments. We have carefully addressed your concerns in our rebuttal and would greatly appreciate it if you could take a moment to review our responses.
We would be happy to provide further clarification or additional evidence if any of our responses remain unclear. Thank you for your time and consideration!