acceptodds
Under review as a conference paper at ICLR 2027

IR³: Index Once, Retrieve, Restore, and Reduce for Multi-Query Long-Video Understanding

Abstract

Popular long videos often receive many questions from different users, yet conventional systems repeatedly perform query-independent visual processing for every question, which leads to substantial redundant computation and inefficient online processing. Existing efficiency approaches mainly follow two directions. Query-conditioned keyframe retrieval improves evidence relevance, but typically introduces additional online scoring and selection latency and may fragment temporal context, omitting event development or procedural order. Visual-token compression, in contrast, reduces downstream inference cost by removing redundant visual tokens while largely preserving task performance, but usually operates only on already selected evidence. Moreover, learning fine-grained token-retention decisions from answer-level supervision creates a difficult credit-assignment problem. We introduce (**I**ndex Once, **R**etrieve, **R**estore, and **R**educe) for efficient multi-query long-video understanding. constructs a reusable, query-independent visual index once per video and uses a hybrid global–local retriever to efficiently locate sparse question-relevant anchors. It then restores their contiguous temporal context and compresses the completed evidence to exactly match the visual-token budget of direct sparse inference. To address diffuse credit assignment, introduces semantic-block budget allocation, enabling more structured and stable token-budget distribution across semantically coherent regions. A causal memory further guides subsequent allocations using evidence retained from earlier visual slices. Across 6,196 questions from three long-video benchmarks, 's retrieval-only path maintains nearly identical accuracy to WFS32 while reducing steady-state online latency from 8.450 to 0.510 seconds. On videos of at least 30 minutes, provides accuracy gains while preserving the same final visual-token budget. Matched-training ablations further show that semantic-block actions provide more stable optimization than token-wise actions under answer-level supervision.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Long-Video Understanding, Video-Language Models, Visual Token Compression, Efficient Multimodal Inference, Reinforcement Learning, Video Retrieval

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_xyMMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper proposes IR^3, a framework for efficient multi-query long-video question answering.

IR^3 has four stages. it first builds a reusable visual index for each video. Then it retrieves query-relevant frames using a hybrid global-local retriever. Next it aggregates the temporal context surrounding the retrieved frames. And finally it compresses the aggregated visual tokens to the original token budget. The compression module uses a learned budget allocation trained with supervised initialization and reinforcement learning.

The paper experimented on LongVideoBench, MLVU, and Video-MME with two 7B vision-language models show modest accuracy improvements over sparse frame retrieval, with larger gains on longer videos. The proposed retriever also achieves substantially lower online latency than WFS32.

Critical strengths
  • The proposed hybrid retriever achieves comparable accuracy to WFS32 while substantially reducing online latency. The paper also provides a detailed latency breakdown and analyzes how offline indexing costs are amortized across queries.
  • The paper identifies a core limitation of sparse frame retrieval, in that isolated frames may miss important temporal context. Aggregating information from nearby frames before compression is an intuitive approach to this problem.
Critical weaknesses
  • The proposed pipeline is considerably more complex than it baselines and involves multiple components that require additional training. However, the empirical gains of the proposed framework are marginal to modest, which seems insufficient considering the complexity of the pipeline. The experiment results do not convincingly establish that the additional complexity is necessary or justified.
  • The paper presents insufficient evidence for the claimed efficiency advantages of the pipeline. Efficiency/latency is one of the main motivations of the method, yet the full IR^3 pipeline is substantially slower than retrieval-only alternatives. From the experiment results it seems that the learned compression stage takes longer than the prefill time it saves when running on a single GPU. The paper also does not report the latency of simpler alternatives such as CLIP32 with precomputed visual features, despite CLIP32 achieving nearly identical accuracy to Hybrid32.
Questions for authors
  • How does IR^3 compare against simpler retrieval and compression pipelines in terms of both accuracy and efficiency, under matched hardware resource?
  • How were the key hyperparameters selected, and how sensitive are the results to these choices? The proposed pipeline contains numerous design choices and hyperparameters. Were these hyperparameter values selected using held-out validation data?
Reviewer_WTbFOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes IR3IR^3, which indexes each video once and reuses it across questions. It first utilizes hybrid retrieval (local and global) to find query-related anchor frames, where the global branch uses CLIP similarity and the local branch uses learned coarse-to-fine window ranking. These anchor frames are expanded with GOP for more robust context preservation. After that, their block-level allocation, trained with RL, finally determines how many visual tokens are kept in each block, so that the final input matches the token budget of direct anchor inference. IR3IR^3 improves performance, especially on longer videos. The method is efficient in one-video many-question scenarios, which are a common and important real-world situation, and largely reduces online latency compared to query-conditioned frame selection.

Critical strengths
  1. IR3IR^3 is effective on long videos, and the paper conducts various experiments.

  2. The motivated situation (one video, many questions) is appropriate and important in the real world.

Critical weaknesses
  1. Weak baselines and marginal performance gain. The main table (Table 1) contains almost no SOTA baselines from other papers. Instead, it compares mostly with ablation-level methods. A comparison with training-based methods, at least with CaCoVID and SCORE mentioned in the paper, seems minimally necessary. Moreover, the performance improvement in Table 1 is marginal, and although the compressor requires both SFT and RL, its gain over the training-free OTT-Vid is very small (Table 5). This makes it hard to judge whether the additional training cost is justified.

  2. Limited novelty and hard to read. The paper’s motivation is intriguing, but the method does not go beyond a simple combination of retrieval and token compression. Expanding key frames with their neighboring frames, similar to their GOP approach, is also common in long-video evidence retrieval. In addition, the paper is hard to read and not self-contained. As an overview, Sec 3.2 contains expressions that require knowledge of later sections, and as a detailed description, it lacks many details. Uncommon expressions such as anchor, spatial token, and zi,j,pz_{i,j,p} are used without clear definition, and their dimensions are not described. These make the paper really hard to understand.

Questions for authors
  1. Are the retriever and the compressor trained jointly or sequentially, and how are anchors obtained when training the compressor? If anchors differ between training and inference, how does this affect the compressor?

  2. The coarse retriever uses temporal coordinates as input. Could it learn a position bias from the LVBench annotations, and how does retrieval perform without the temporal term?

Reviewer_SfzwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

IR³ builds a reusable video index, retrieves sparse anchors, and restores their surrounding temporal context. A learned compressor allocates token budgets across semantic blocks, reducing the expanded evidence to the original token count.

Critical strengths

Reusable indexing addresses repeated visual processing across questions, and the retrieval-only path achieves accuracy comparable to WFS32 with a roughly 16.6× steady-state online speedup. Restore–Reduce further improves accuracy over direct sparse retrieval by 2.22 percentage points on videos lasting at least 30 minutes, while preserving the final token budget.

Critical weaknesses
  1. The learned compressor provides modest gains over random retention and OTT-Vid, while the downstream QA comparison in Table 4 uses a single training seed.
  2. CLIP32 achieves pooled accuracy close to Hybrid32, but its online latency under the same reusable-index setup is not reported.
  3. Latency measurements cover only four videos with preloaded indices, and index storage size and peak memory are not reported. Broader measurements are needed to clarify the resource requirements and applicability of the reported speedups.
Questions for authors
  1. Do the learned compressor’s accuracy gains over random retention and OTT-Vid remain consistent across training seeds?
  2. How does cached CLIP32 compare with Hybrid32 in online latency under the same indexing and answering setup?
  3. What are the index storage size and peak memory requirements as video duration increases?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.