- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes READ FIRST, a framework for long-video question answering. It constructs multi-scale memory from visual captions and speech transcripts, supplemented by semantic records and an index linking timestamps to video frames. At inference time, the system first attempts to answer directly from a complete transcript; otherwise, it gathers evidence through iterative BM25-based retrieval and model-based reranking. A second-opinion gate derives two answers from the same evidence and triggers targeted visual inspection only when they disagree. The resulting visual observations are converted into text and incorporated into the evidence, with an additional diagnostic step for unresolved cases. Experiments on five long-video benchmarks report a macro-average accuracy of 74.15% with GPT-5, compared with 72.88% before, and a macro-average of 93.7% of questions resolved without online visual inspection. Ablations and cost analyses examine the contributions of memory construction, retrieval, and selective inspection.
Critical strengths
- A practical and solid strategy for allocating evidence. Rather than adding visual input to every query, the framework reuses textual memory and reserves targeted inspection for questions whose answers disagree under identical evidence. With GPT-5, reading alone achieves 72.88% macro-average accuracy, while the full system reaches 74.15%, with most of questions resolved without online visual inspection.
- Demonstrates that conditional visual invocation is key. The paper compares targeted visual inspection with a third text-only answer generated from the same evidence. On the escalated subset, this additional text-only derivation reaches 43.0% accuracy, whereas visual inspection raises accuracy to 51.1%. These controls strengthen my assessment because they show that visual inspection is not uniformly helpful, and that conditional, selective invocation is what matters, thereby addressing two central alternative explanations: that improvements merely come from additional model calls, or that visual inspection would help equally without selective gating.
Critical weaknesses
- Limited methodological novelty. Converting video into textual descriptions and reasoning over them was already an established approach by 2024. LLoVi [1] uses densely sampled clip captions and multi-round summarization for question answering; VideoTree [2] organizes query-relevant visual descriptions into a hierarchical representation; and DrVideo [3] converts videos into textual documents and iteratively retrieves and augments missing evidence. The paper's core contribution is already a mature pipeline; it merely provides transcription and hierarchical organization tools on top of it, without offering any unique understanding or contribution to the field of video understanding.
- The claim that the marginal value of visual input varies with duration does not necessarily hold. The central ablation supplies the complete, untruncated language track but restricts visual input to sixteen frames. This is a legitimate bounded-budget setting, yet the resulting decline in the marginal value of visual input does not establish that visual evidence generally becomes less useful as videos grow longer. Different duration groups also contain different videos and questions, so increasing text-only accuracy cannot be interpreted as a causal effect of duration. More ablation experiments should be conducted to verify this conclusion. The diminishing effectiveness of visual information may well be because the model is not sufficiently fitted on such data, rather than because visual information itself is not valuable.
[1] Ce Zhang et al. A Simple LLM Framework for Long-Range Video Question-Answering (LLoVi). EMNLP 2024.
[2] Ziyang Wang et al. VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videos. CVPR 2025.
[3] Ziyu Ma et al. DrVideo: Document Retrieval Based Long Video Understanding arXiv:2406.12846
Questions for authors
Please look the Critical Weaknesses. I will make my final decision based on the other reviewers' comments and the rebuttal.