- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper proposes IR^3, a framework for efficient multi-query long-video question answering.
IR^3 has four stages. it first builds a reusable visual index for each video. Then it retrieves query-relevant frames using a hybrid global-local retriever. Next it aggregates the temporal context surrounding the retrieved frames. And finally it compresses the aggregated visual tokens to the original token budget. The compression module uses a learned budget allocation trained with supervised initialization and reinforcement learning.
The paper experimented on LongVideoBench, MLVU, and Video-MME with two 7B vision-language models show modest accuracy improvements over sparse frame retrieval, with larger gains on longer videos. The proposed retriever also achieves substantially lower online latency than WFS32.
Critical strengths
- The proposed hybrid retriever achieves comparable accuracy to WFS32 while substantially reducing online latency. The paper also provides a detailed latency breakdown and analyzes how offline indexing costs are amortized across queries.
- The paper identifies a core limitation of sparse frame retrieval, in that isolated frames may miss important temporal context. Aggregating information from nearby frames before compression is an intuitive approach to this problem.
Critical weaknesses
- The proposed pipeline is considerably more complex than it baselines and involves multiple components that require additional training. However, the empirical gains of the proposed framework are marginal to modest, which seems insufficient considering the complexity of the pipeline. The experiment results do not convincingly establish that the additional complexity is necessary or justified.
- The paper presents insufficient evidence for the claimed efficiency advantages of the pipeline. Efficiency/latency is one of the main motivations of the method, yet the full IR^3 pipeline is substantially slower than retrieval-only alternatives. From the experiment results it seems that the learned compression stage takes longer than the prefill time it saves when running on a single GPU. The paper also does not report the latency of simpler alternatives such as CLIP32 with precomputed visual features, despite CLIP32 achieving nearly identical accuracy to Hybrid32.
Questions for authors
- How does IR^3 compare against simpler retrieval and compression pipelines in terms of both accuracy and efficiency, under matched hardware resource?
- How were the key hyperparameters selected, and how sensitive are the results to these choices? The proposed pipeline contains numerous design choices and hyperparameters. Were these hyperparameter values selected using held-out validation data?