- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies VideoRAG for long videos and focuses on how video chunks should be represented for retrieval and generation. It introduces V-RAGBench, a benchmark of 2,100 <query, evidence chunk, answer> triplets over 216 hour-scale videos. It also proposes CARVE, a training-free method that retrieves candidates from multiple modality–granularity configurations, reranks them at the chunk level, and passes each retrieved chunk to the generator using its selected representation.
Critical strengths
- V-RAGBench is proposed so that retrieval and generation can be evaluated separately. The benchmark uses multiple filtering stages and human verification to reduce shortcut answerability and spurious evidence.
- CARVE allows different retrieved chunks to use different modality–granularity configurations. The method is training-free, easy to integrate into existing retrieval pipelines, and shows consistent gains across multiple baselines and generator backbones.
Critical weaknesses
-
Although V-RAGBench assumes that each question can only be answered using its annotated evidence chunk, only 97.1% of non-evidence windows were judged unanswerable. More importantly, the paper does not clarify whether the problematic samples identified during human verification were removed or corrected. These issues should be addressed, particularly for the test set, to ensure reliable evaluation.
-
The empirical answerability filter removes questions that Gemini-2.5-Flash fails to answer even when given the ground-truth evidence. While this helps ensure answerability, it may also exclude difficult but valid questions. As a result, the benchmark's question distribution may be influenced by the capabilities of the filtering model.
Questions for authors
- Datasets such as NExT-GQA already provide questions, answers, and temporal evidence annotations. Could the authors clarify what V-RAGBench adds beyond the unique-evidence constraints and longer videos?
- CARVE uses the configuration selected during reranking directly for generation. However, a representation that is effective for retrieving a chunk may not be the best one for answering the question. Have the authors considered selecting the generation representation separately?
My rating may change depending on how the authors address my concerns in their rebuttal.
5 comments
W2.
W2. The empirical answerability filter removes questions that Gemini-2.5-Flash fails to answer even when given the ground-truth evidence. While this helps ensure answerability, it may also exclude difficult but valid questions. As a result, the benchmark's question distribution may be influenced by the capabilities of the filtering model.
We appreciate the reviewer’s important observation. We agree that filtering with a specific model could exclude difficult yet valid questions. However, we would like to respecfully emphasize two key aspects of our setup that prevent this filter from making the benchmark overly easy or model-dependent.
Firstly, in the Empirical Answerability filter, Gemini-2.5-flash inspects only the isolated, short ground-truth event clip (median 8.6s). Because localized verification on a single short clip is far simpler than long-video RAG, the filter primarily screens out visually ambiguous or ill-posed candidates rather than reasoning-heavy questions.
Secondly, there is still a high difficulty ceiling under full RAG evaluation. Passing this localized check leaves substantial challenge at evaluation time, where models must retrieve evidence from 1–9 hour videos and reason amidst top- distractors. In Table 2, end-to-end pass rates remain at 0.318–0.368 even for CARVE. Moreover, in our new GT-insertion experiment (see our response to Reviewer M7Wp, W1), even when the GT chunk is guaranteed in the top-5 context, Qwen3-VL-8B achieves 0.322 and Gemini-3.8-Flash reaches 0.630, demonstrating a large gap yet to be closed.
We will revise the manuscript to clarify this distinction between localized verification and full RAG evaluation, and note how future extensions can audit rejected candidates to incorporate harder perceptual questions.
Q1.
Q1. Datasets such as NExT-GQA already provide questions, answers, and temporal evidence annotations. Could the authors clarify what V-RAGBench adds beyond the unique-evidence constraints and longer videos?
We thank the reviewer for the opportunity to clarify how V-RAGBench complements grounded video QA benchmarks such as NExT-GQA. While scaling to hour-long videos and enforcing unique evidence are foundational for moving beyond single-pass in-context processing, V-RAGBench introduces two additional benchmark design properties specifically tailored for VideoRAG evaluation.
First, whereas NExT-GQA is formulated for in-context temporal grounding where a short clip is fed entirely into a model to predict a continuous time interval , V-RAGBench is structured around a discrete retrieve-then-generate pipeline. This allows researchers to separately measure first-stage candidate retrieval and reranking (via Recall@ and nDCG@) and downstream generation under retrieved top- distractors.
Second, V-RAGBench is constructed to prevent retrieval and generation shortcuts. During query generation, each question is required to embed contextual situation anchors rather than explicit temporal cues (Section 4), forcing retrievers to match visual and semantic content. Moreover, unlike multiple-choice setups, V-RAGBench pairs open-ended reference answers with blind shortcut filtering and empirical answerability checks so that downstream accuracy reflects grounded generation rather than option guessing or language priors.
We will cite NExT-GQA and clearly articulate these complementary evaluation goals in Section 2 (Related Work).
We sincerely thank the reviewer for the positive evaluation of our work and for providing important feedback on our benchmark and method design. Your comments motivated us to run additional experiments for W1 and Q2 to deepen our analysis, and we have done our best to address each of your points as thoroughly as possible below.
W1.
We thank the reviewer for highlighting this point. We fully share your view that maintaining strict evidence uniqueness is critical for reliable evaluation. While there is a remaining gap to 100%, we respectfully note that the 97.1% figure was measured exhaustively across all 10,206 non-evidence window–query pairs in the test videos, which we believe provides a solid empirical basis for evidence localization in hour-scale recordings.
Having said that, to address your concern directly and ensure complete test-set reliability, we re-evaluated all 10 methods after strictly excluding the test queries associated with the flagged windows from the human audit.
Additional results after excluding the flagged test queries
All metrics change by at most ±0.008 compared to Table 2, and all method rankings remain completely unchanged. We are deeply grateful to you for highlighting this point, as it truly helped us address a critical detail and make our evaluation much more rigorous. We will include this sensitivity analysis and provide the cleaned test split in the revision.