acceptodds
Withdrawn from ICLR 2027

Read First, Look on Doubt: A Text-First Framework for Long-Video Question Answering

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal large language models have advanced long-video question answering chiefly by looking harder, through longer frame contexts, compressed visual tokens and multimodal memory agents, all treating the collapse of visual coverage as the problem to solve. We report two observations that point the other way. As videos lengthen, the language of the footage, its speech, narration and on-screen text, answers a growing share of what is asked on its own, and accuracy from the language track alone grows steadily stronger with duration while accuracy from visual input collapses. Visual information added on top of the text in turn loses its marginal value, which falls from points on the shortest clips to at the longest tier. In response, we propose ReadFirst, a text-first framework that inverts the standard allocation. It distils every channel of a video into a multi-scale textual memory offline and answers by reading at query time, escalating to a targeted visual inspection only when two derivations of the same answer disagree, thereby spending perception only where the text runs out. Extensive experiments demonstrate that ReadFirst achieves state-of-the-art performance, reaching an average accuracy of 74.1 across five long-video benchmarks while resolving over 90% of questions in text alone.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Long-Video Question Answering, Multimodal Large Language Models, Video Understanding, Agent Memory

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vBYGOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes READ FIRST, a framework for long-video question answering. It constructs multi-scale memory from visual captions and speech transcripts, supplemented by semantic records and an index linking timestamps to video frames. At inference time, the system first attempts to answer directly from a complete transcript; otherwise, it gathers evidence through iterative BM25-based retrieval and model-based reranking. A second-opinion gate derives two answers from the same evidence and triggers targeted visual inspection only when they disagree. The resulting visual observations are converted into text and incorporated into the evidence, with an additional diagnostic step for unresolved cases. Experiments on five long-video benchmarks report a macro-average accuracy of 74.15% with GPT-5, compared with 72.88% before, and a macro-average of 93.7% of questions resolved without online visual inspection. Ablations and cost analyses examine the contributions of memory construction, retrieval, and selective inspection.

Critical strengths
  • A practical and solid strategy for allocating evidence. Rather than adding visual input to every query, the framework reuses textual memory and reserves targeted inspection for questions whose answers disagree under identical evidence. With GPT-5, reading alone achieves 72.88% macro-average accuracy, while the full system reaches 74.15%, with most of questions resolved without online visual inspection.
  • Demonstrates that conditional visual invocation is key. The paper compares targeted visual inspection with a third text-only answer generated from the same evidence. On the escalated subset, this additional text-only derivation reaches 43.0% accuracy, whereas visual inspection raises accuracy to 51.1%. These controls strengthen my assessment because they show that visual inspection is not uniformly helpful, and that conditional, selective invocation is what matters, thereby addressing two central alternative explanations: that improvements merely come from additional model calls, or that visual inspection would help equally without selective gating.
Critical weaknesses
  1. Limited methodological novelty. Converting video into textual descriptions and reasoning over them was already an established approach by 2024. LLoVi [1] uses densely sampled clip captions and multi-round summarization for question answering; VideoTree [2] organizes query-relevant visual descriptions into a hierarchical representation; and DrVideo [3] converts videos into textual documents and iteratively retrieves and augments missing evidence. The paper's core contribution is already a mature pipeline; it merely provides transcription and hierarchical organization tools on top of it, without offering any unique understanding or contribution to the field of video understanding.
  2. The claim that the marginal value of visual input varies with duration does not necessarily hold. The central ablation supplies the complete, untruncated language track but restricts visual input to sixteen frames. This is a legitimate bounded-budget setting, yet the resulting decline in the marginal value of visual input does not establish that visual evidence generally becomes less useful as videos grow longer. Different duration groups also contain different videos and questions, so increasing text-only accuracy cannot be interpreted as a causal effect of duration. More ablation experiments should be conducted to verify this conclusion. The diminishing effectiveness of visual information may well be because the model is not sufficiently fitted on such data, rather than because visual information itself is not valuable.

[1] Ce Zhang et al. A Simple LLM Framework for Long-Range Video Question-Answering (LLoVi). EMNLP 2024.

[2] Ziyang Wang et al. VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videos. CVPR 2025.

[3] Ziyu Ma et al. DrVideo: Document Retrieval Based Long Video Understanding arXiv:2406.12846

Questions for authors

Please look the Critical Weaknesses. I will make my final decision based on the other reviewers' comments and the rebuttal.

Reviewer_UNSLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents READFIRST, a text-first framework for long-video question answering. It processes each video offline into a multi-scale textual memory that combines speech and visual descriptions, and answers questions by retrieving from this memory. The approach is motivated by the observation that, under a fixed frame budget, the added value of visual input decreases as video duration increases. Experiments on five long-video benchmarks report an average accuracy of 74.1 with GPT-5. Most questions are resolved without query-time visual inspection, while the visual stages provide targeted support for the smaller set of uncertain cases.

Critical strengths
  1. The motivation is convincing. The paper starts from a simple but important observation: with a fixed frame budget, visual input becomes less useful as videos get longer. This gives a clear reason for reconsidering the usual pipeline. Based on it, the overall pipeline is carefully designed. READFIRST distills speech and visual content into a multi-scale textual memory offline and answers questions by reading this memory at query time. When the retrieved evidence does not support a stable answer, it returns to a small number of relevant video segments for closer inspection. This staged design handles text-dominant questions efficiently while preserving the ability to recover finer visual evidence when necessary.
  2. The experiments are extensive, covering five benchmarks and two reasoning models. The paper compares against several types of baselines and includes ablations on the main components, providing solid empirical support for the proposed design.
Critical weaknesses
  1. Overall, I find the novelty a little bit limited. WorldMM already lets an agent choose between textual and visual memories as it gathers evidence. READFIRST makes this process more restrictive: it reads text first and checks the video when two answers disagree. This is a useful design choice, but I see it as an incremental change to the control policy rather than a substantial methodological advance.
  2. Appendix E reports the number of calls needed to build the memory, but not the token usage or actual construction time. Since different construction steps may involve very different input and output lengths, reporting the token usage and construction time per video would provide a more informative measure of the offline cost.
Questions for authors
  1. Please see the Weaknesses.
  2. Could the authors provide the main prompt templates used in the pipeline? This would make the system easier to understand and reproduce.
  3. The paper uses BM25 as a lightweight choice that avoids query-time GPU usage. How does this choice compare with embedding-based or hybrid retrieval in terms of both QA accuracy and efficiency? A comparison between BM25 and other query methods would therefore be useful for understanding the trade-off between retrieval accuracy and query-time efficiency.
  4. I wonder how sensitive the method is to errors or missing information in the offline textual memory. For example, if the ASR is wrong or the captioner misses an important visual detail, the system may lose both the evidence needed to answer the question and the cues needed to find the relevant video segment. This could be an issue because the answering, gating, and segment-selection steps all depend on this memory.
Reviewer_e4r2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents ReadFirst, a text‑first framework for long‑form video QA. The authors empirically observe that under fixed query‑time visual budget, text‑only performance improves with video length while vision‑only performance degrades; adding frames can even hurt accuracy for very long videos. ReadFirst distils full‑video visual and audio information into multi‑scale textual memory offline. During online inference, it answers primarily via text‑only iterative BM25‑based retrieval. Its core novelty is a second‑opinion gate: visual inspection is only triggered when two independent answer derivations over identical textual evidence disagree. Visual observations are converted to text and appended for reasoning. Evaluated on five long‑video benchmarks, ReadFirst achieves SOTA accuracy. Ablations validate each component, and the gate outperforms confidence‑based triggers with zero calibration. It removes query‑time GPU dependency but carries offline‑processing and transcript‑quality limitations.

Critical strengths

This work delivers a compelling paradigm shift from perception‑first to text‑first long‑video QA, well‑supported by controlled modality ablations showing diminishing, even negative marginal utility for query‑time visual inputs under fixed compute budgets. It introduces multiple practical innovations: traceable multi‑scale textual memory removing query‑time GPU overhead, a calibration‑free second‑opinion gate that conditionally invokes visual inspection on answer disagreement, and verdict‑free visual inspection converting observations back to text for unified reasoning. Evaluations across five benchmarks include strong matched baselines and thorough ablations validating each component. Quantified results show over 90% of questions can be solved by text alone, with visual compute precisely allocated only to hard ambiguous cases. The framework works for both closed‑source and open reasoning models, offering an efficient practical blueprint for long‑video QA research.

Critical weaknesses
  1. The framework heavily relies on the quality of offline ASR and frame‑based captioning, yet systematic analysis on noise propagation from ASR/caption hallucinations is missing. Evaluation under challenging domains such as low‑SNR audio or OCR‑intensive videos is also absent, leaving real‑world domain robustness insufficiently validated.

  2. Though the paper highlights the benefit of removing query‑time GPU usage, end‑to‑end quantified metrics for offline pre‑processing (ASR + dense captioning), storage overhead, and amortized per‑question latency are not fully reported. These figures are critical to assess the real‑world practical viability of this one‑time‑build, multiple‑query paradigm.

  3. While a broad set of baselines is included, there is a lack of quantitative, controlled comparison against hierarchical‑caption‑with‑verification methods (e.g., DeVi [1]). Moreover, conceptual discussion is thin for alternative competing paradigms including token‑adaptive streaming compression (AdaCM2 [2]) and agentic plan‑ground‑verify frameworks (VideoMind), without clarifying when text‑first should be preferred over these alternatives.

  4. Important mechanics are not fully concretized in the paper, including exact decoding hyper‑parameters for generating the second‑opinion answer, concrete implementation of the “model‑channel” time‑window proposal, and the diagnostic‑branch routing logic inside Stage 3. This reduces reproducibility of the core gate and visual‑escalation modules.

[1] Question-answering dense video events [2] AdaCM2: On Understanding Extremely Long-Term Video with Adaptive Cross-Modality Memory Reduction [3] VideoMind: A chain-of-LoRA agent for temporal-grounded video reasoning

Questions for authors

See Weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.