acceptodds
Under review as a conference paper at ICLR 2027

Coarse Indexing, Fine Evidence: Decoupling Temporal Granularity in Long-Video RAG

Abstract

Graph-based retrieval-augmented generation (RAG) provides a scalable paradigm for long-video understanding, but existing systems typically inherit a fixed temporal granularity from video segmentation when constructing their retrieval index. We argue that this design unnecessarily couples indexing granularity with evidence granularity: coarse representations can often suffice for locating relevant temporal regions, while fine-grained evidence remains important for downstream reasoning. We propose Density-Aware Graph Construction (DAGC), a training-free approach that decouples a query-independent coarse retrieval index from the original fine-grained evidence space. DAGC constructs a compact, density-adaptive graph index by merging visually redundant neighboring chunks, while preserving mappings to the original temporal units. Retrieved coarse regions are subsequently expanded back to the original chunk granularity for fine-grained evidence refinement and answer generation. Experiments on MLVU, VideoMME, and LongVideoBench show that DAGC retains only about 40–50% of the original graph nodes and achieves – end-to-end wall-clock acceleration while preserving approximately 99% of the original QA performance. The gains transfer across different LVLM backbones and video RAG pipelines, suggesting that long-video RAG need not maintain the same temporal granularity for indexing and evidence reasoning.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
long video understanding, graph-based RAG, efficient graph construction

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_mj3mOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes DAGC, a training-free coarse-to-fine indexing strategy that reduces the cost of building video graphs. On top of Vgent, DAGC merges visually redundant adjacent chunks into a lighter graph for offline construction, while keeping a mapping to the original chunks. After retrieval on this coarse graph, the selected regions are mapped back to the original chunks for fine-grained downstream reasoning. Experiments on VideoMME, MLVU, and LVB show reduced construction time with largely maintained performance. The method also transfers to another LVLM family and another video RAG framework.

Critical strengths
  1. Extensive experiments across various VLM backbones and RAG frameworks support the method’s robustness.
  2. The motivation of decoupling relevant-region localization from detailed downstream reasoning is convincing and timely.
Critical weaknesses
  1. The paper is hard to read and not self-contained. Although DAGC operates on top of Vgent, many parts of the method are not described in detail, and they are not covered in the appendix. For example, there is no description of which VLM is used for entity extraction, no exact formulation of RsR_s and Rs+R_s^+, and no clarification of whether cic_i is raw pixels or visual features after the vision encoder or adapter. In addition, Table 5 lacks sufficient explanations of SN, RR, and TE. These omissions make the paper difficult to understand and reproduce.

  2. Limited novelty and insufficient justification. Coarse-to-fine indexing is reasonable, but it reads as a simple application, adapting ideas already explored in many other areas. The paper is not convincing on why coarse indexing is sufficient for video RAG and how the resulting accuracy–efficiency trade-off can be minimized, since most explanations rely on empirical results rather than logical justification. The efficiency claim is also not fully supported. There is no complexity analysis, and it is unclear whether the frames of merged nodes overlap with the original chunks and whether vision encoder features are reused. If they are not reused, it is unclear whether this additional cost is included in the offline time. Finally, the interpretation of why performance drops excessively only on VideoMME is insufficient. Attributing it simply to longer videos is not convincing, since this does not hold for LVB.

Questions for authors
  1. Is cic_i raw pixels or visual features after the vision encoder? If features are used, which encoder is used, and are the encoded features reused when re-sampling frames for merged nodes?

  2. What causes the larger drop on VideoMME compared to LVB and MLVU? An analysis of failure cases, such as whether the key evidence was merged into coarse nodes, would help clarify this.

1 comment

Thank you for the detailed rebuttal. The clarifications on the efficiency analysis and Q1–Q2 helped me better understand the paper. However, I still have concerns regarding (1) the presentation, (2) the justification of coarse indexing, and (3) the novelty.

1. Presentation.

Most of my original points concerned missing descriptions, and the clarifications in the rebuttal (e.g., the definitions of RsR_s and Rs+R_s^+, the graph construction criterion, and the BGE-large-en-v1.5 embedding model) are not described in the paper at all. Since ICLR allows revisions during the rebuttal period, I recommend reflecting them in the manuscript. The paper should also state which retrieval components are inherited from Vgent and which are modified or newly introduced. In addition, please clarify whether semantic extraction for the other backbones (Qwen2.5-VL-3B, Qwen2-VL-7B, InternVL3.5-8B) also uses Qwen2.5-VL-7B-Instruct.

2. Justification of coarse indexing.

Table 5 does not directly show why coarse indexing is sufficient. Since RR jointly covers recovery and reranking, the gain from 60.6 to 63.1 cannot be attributed to fine-grained recovery alone. A more direct analysis would be to quantitatively compare retrieval performance between fine-grained (Vgent) and coarse-grained retrieval.

3. Novelty.

The paper does not discuss the challenges of converting a fine-grained graph index into a coarse one, nor does it provide detailed experiments on the related design choices (e.g., adapting graph construction for merged nodes, or using vision-encoder features or a lightweight encoder instead of raw-pixel descriptors). Without such considerations, DAGC reads as simply inserting a coarse-indexing step into Vgent, which feels more like an engineering modification.

Reviewer_ZH27OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper argues existing graph rag methods for video understanding index a fixed temporal segment for retrieval, but the video indexing should pair with evidence granularity. To this end, they propose ensity-Aware Graph Construction (DAGC), which constructs a density-adaptive graph index by merging visually redundant neighboring chunks. Experiments on three long video understanding benchmarks show that DAGC keeps only about 40–50% of the original graph nodes and achieves 1.3–1.7× speed up, while preserving 99% of the original QA performance.

Critical strengths
  1. The paper identify a critical problem in existing method and the video segment should be partitioned and merged based on their evidence density.
  2. The node merging speed up the inference time while having minimum impact to the baseline's accuracy.
  3. The method is straight forward and easy to follow.
Critical weaknesses
  1. The merging of several chunks will result in a node with at most 3 times longer, but still sample with 64 frames for graph construction and entity extraction. There is no analysis for different questions that this sparse sampling will make very fine-grained details be missed.
  2. There is only one baseline and the base models (qwen2/2.5) are quite old. The paper didn't compare with other video rag methods.
  3. The technical contribution over the baseline is the chunk merging based on visual similarity, which is quite limited. Since the visual similarity based token merging, token reduction, frame reduction has been used in a lot of existing works, like MovieChat, FastVID, LongVU, MA-LMM, VideoTree.
  4. There is no comparison with other visual merging methods, nor ablation on which visual encoder (embedding space) is the best. Given the merging is the only contribution, the paper should give more insight and experiments on this.
Questions for authors
  1. For those merged chunk, the number of sampled frames is still 64. Would this sparse sampling negatively affects some fine-grained questions that needs to localized specific frame. If it is sparsely sampled and missed, it might also be missed during retrieval.
  2. Which visual encoder is used for visual feature extraction and chunk merging.
  3. Have you tried ablations on different visual encoders? Since the chunk merging is the only contribution, the paper should provide insight on which embedding space is the most effective one.
  4. Can you compare with other visual similarity-based merging method, for example, different embedding space, different similarity calculation, different merging methods (like visual clustering in VideoTree).
  5. It seems there is no much accuracy difference between merge at most 3 chunks or more, why choose 3 instead of 5, which will make it more cost effective.
  6. VideoRAG is also a graph-based video rag method, why it doesn't appear in Table 1?
Reviewer_BdDfOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Density-Aware Graph Construction (DAGC) for efficient long-video RAG. DAGC merges visually similar neighboring chunks before semantic extraction, builds a compact graph with a fixed frame budget per node, and recovers the original chunks after retrieval for downstream reasoning. The method substantially reduces indexing cost while largely preserving average QA accuracy.

Critical strengths
  1. The method offers an interesting way to reduce graph-construction cost.

  2. The experiments are fairly comprehensive. Experiments cover three benchmarks, multiple Qwen backbones, InternVL, and two RAG frameworks. Comparisons with uniform and random merging under the same node budget also indicate that content-aware merging contributes to accuracy preservation.

  3. Experimental results demonstrate substantial computational savings, retaining approximately half the indexing units and achieving wall-clock speedups of 1.3–1.7×.

Critical weaknesses
  1. The effects of evidence recovery and reranking are not evaluated separately. Removing RR reduces accuracy from 63.1 to 60.6, but this removes both original-chunk recovery and reranking. It is therefore unclear how much of the improvement comes from recovering fine-grained evidence. Also, DAGC without compression scores 62.6, compared with 63.3 for Vgent, despite retaining all nodes. The authors could evaluate recovery and reranking separately and explain the difference between the uncompressed variant and Vgent.

  2. The distinction from existing coarse-to-fine methods is not clear enough. VideoTree and VideoRAG already acquire finer evidence after coarse localization. DAGC instead focuses on query-independent adaptive merging of temporal indexing units before semantic extraction. A more explicit comparison of these mechanisms would help clarify the paper’s contribution.

Questions for authors

See above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.