- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces RCES, a training-free framework for selecting evidence for long-video question answering. Its main contribution is to formulate the selection of unequal-length textual evidence as knapsack-constrained optimization. The objective combines question relevance, temporal coverage, source coverage, and semantic representativeness, and is optimized using density greedy with a best-singleton safeguard. Video frames are selected separately through uniform anchors and question-relevant frame packets. Experiments across three benchmarks and five frozen VLM backbones show competitive accuracy with reductions in evidence volume or latency, depending on the setting. Controlled ablations examine the contributions of the text objective and frame allocator.
Critical strengths
- A useful formulation of a practical evidence-selection problem. The paper clearly distinguishes retrieving potentially relevant evidence from deciding what should actually occupy the final prompt. Accounting for unequal text lengths and the marginal value of an evidence set is more appropriate than controlling only the number of retrieved items. The resulting selector requires neither model training nor additional calls to the answering VLM, making it a practical addition to retrieval-based video understanding pipelines.
- Meaningful accuracy–efficiency trade-offs with useful controlled ablations. Under the common text budget on VideoMME, RCES uses approximately 24% less textual evidence than TV-RAG, while achieving slightly higher measured accuracy and substantially lower reported latency across three backbones. Importantly, the paper also includes fixed-pool, fixed-frame comparisons and individual objective-term removals, rather than relying only on comparisons against models without retrieved text. These experiments provide concrete support for the proposed selection strategy.
Critical weaknesses
-
The text selector’s independent benefit needs broader validation. The controlled text-selection comparisons use only Qwen2.5-VL on VideoMME at with uniform frames, leaving unclear whether the gains persist in the final packet-based pipeline or other settings. Moreover, 88.6% of the distilled MLVU pools already fit within the budget, so most examples do not require the selection trade-offs central to the paper. Please extend the fixed-pool comparisons to the final frame configuration and at least one additional benchmark or backbone, separately report examples whose candidate pools exceed the budget, and provide paired confidence intervals for the selector’s gains.
-
The reported latency advantage is difficult to attribute to evidence allocation. On LLaVA-Video/VideoMME, Video-RAG takes 43.3 seconds with 472 evidence units under the common cap, but only 28.7 seconds with its larger native input of 1,623 units. This unexplained pattern makes it difficult to distinguish savings from evidence allocation from differences in preprocessing, caching, or implementation. Please clarify the timing boundaries and caching protocol across methods, explain this discrepancy, and provide comparable breakdowns for evidence extraction, retrieval/selection, and final VLM inference under matched hardware and cache conditions.
-
The accuracy–cost evaluation relies too heavily on a single operating point. The main comparisons use either a common 800-unit cap or competitors’ native text footprints, while the budget sweep evaluates only RCES. These results do not establish how the methods compare across resource limits. Furthermore, the character-based cost measure is a proxy: its reported correlation with tokenizer counts does not establish equivalent actual input costs across methods. Please specify how each baseline is capped and compare RCES with the strongest retrieval baselines across several budgets, reporting accuracy against measured input tokens and end-to-end latency with the same backbone and frame budget. This would establish whether the advantage extends beyond the selected 800-unit setting.
-
Generalization to newer VLMs remains untested. The evaluation is limited to earlier-generation backbones, including Qwen2.5-VL, LLaVA-Video, and InternVL2.5. This leaves unclear whether RCES’s accuracy–efficiency gains persist on newer models: the benefit of auxiliary text and the savings from input reduction may change with the backbone’s multimodal capabilities and inference architecture. Please compare RCES with the strongest retrieval baselines on recent, comparably sized models, such as Qwen3.5-9B and InternVL3.5-8B.
Questions for authors
see weakness.
6 comments
Part 2
TV-RAG has higher measured extraction/cache-access time (11.82 vs 0.05 s), retrieval/selection time (5.79 vs 1.29 s), and final VLM inference time (31.58 vs 10.19 s). RCES cache access is included in its retrieval/selection time. These stage measurements are not additive. Against Video-RAG, which caches ASR and OCR like RCES, the situation differs: RCES is faster overall because of its lighter frame preprocessing (5.8 vs 21.3 s), while Video-RAG's selection and final VLM call are shorter. We therefore do not attribute the latency gap to Video-RAG to evidence allocation, and we note this as a limitation of the latency comparison. TV-RAG's stage timers are recorded separately; their sum (49.2 s) exceeds its end-to-end latency.
Where the time goes. Per question, RCES does three simple steps: it scores the frames with CLIP (5.79 s), reads the cached speech and on-screen text and selects from it (1.29 s in total, of which the cache read takes 0.05 s and the knapsack selection 0.002 s), and makes one final VLM call (10.19 s), for 17.27 s in total (App. C.7, Table 11). The latency advantage therefore comes from this pipeline design (cached evidence, lightweight selection, and a single VLM call), not from the knapsack optimization itself, which accounts for a negligible share of the time.
W3: Comparison Across Operating Points
How each baseline is capped. All retrieval methods are limited by the same text budget , measured with the same cost proxy as RCES (, Eq. 1). Each baseline keeps its own retrieval, ranking, and preprocessing unchanged; only its final evidence is limited. Baselines add their retrieved items in their native order: Video-RAG stops at the first item that does not fit, and TV-RAG cuts its concatenated evidence at the budget, so its last item may be truncated. RCES instead skips items that do not fit and continues selecting. Video-RAG nevertheless uses a comparable mean evidence cost: under it uses 451 units versus 467 for RCES (Table 1). The backbone, frame count (), hardware, and text budget are matched across methods.
Baselines across budgets. We evaluated the strongest retrieval baselines at additional budgets on VideoMME (Qwen2.5-VL-7B, 64 frames, same budget limits and cost definition):
| Method | Paired test between budgets | |||
|---|---|---|---|---|
| Video-RAG | 62.15 | 61.00 | 64.41 | 500 vs 800: (n.s.); 800 vs 1000: |
| TV-RAG | 63.63 | 64.04 | pending | 500 vs 800: (n.s.) |
| RCES | 64.19 | 65.33 | 65.00 | 500 vs 800: ; 800 vs 1000: (n.s.) |
Among the completed runs, RCES has the highest measured accuracy at every budget (App. C.4, Table 10). Video-RAG's difference between 500 and 800 is not significant, and it improves only at , where RCES remains higher (65.00 vs 64.41). The TV-RAG run at is still in progress and will be added once complete.
Accuracy against measured input tokens. To check the cost proxy against actual textual evidence tokens, we counted the evidence tokens sent to the model with the Qwen2.5-VL tokenizer, applied identically to every method (App. A.2, Table 4):
| Method | Accuracy (%) | Mean evidence tokens | Cost units (Eq. 1) | |
|---|---|---|---|---|
| RCES | 500 | 64.19 | 347 | 319 |
| Video-RAG | 500 | 62.15 | 370 | 321 |
| TV-RAG | 500 | 63.63 | 362 | 347 |
| RCES | 1000 | 65.00 | 566 | 554 |
| Video-RAG | 1000 | 64.41 | 594 | 558 |
At these operating points, the mean token counts are close to the cost units for every method (1.0–1.2 tokens per unit), so the methods receive comparable evidence volumes under the common budget. At each budget, RCES has the highest measured accuracy with the fewest mean evidence tokens.
Latency across budgets. Per-question latency, shared extraction excluded (same backbone, frames, and hardware; seconds per question; App. C.4, Table 10):
| Method | |||
|---|---|---|---|
| Video-RAG | 30.4 | 25.1 | 30.3 |
| TV-RAG | 19.3 | 42.4 | pending |
| RCES | 14.3 | 17.3 | 18.6 |
Among the completed runs, RCES is the fastest at every budget, and its latency grows only moderately with the budget (14.3 to 18.6 s). Baseline timings were obtained in separate runs with their native preprocessing and caching policies; these measurements do not isolate the effect of changing the text budget. The TV-RAG run at is still in progress and will be added once complete.
Part 3
W1 (continued): Final Frame Configuration and Second Backbone
We extended the fixed-pool comparison to the final configuration with frame packets (16 anchors + 12×4 = 64 frames), on Qwen2.5-VL and on a second backbone, LLaVA-Video-7B (VideoMME, , identical candidate pools). Entries are overall / long-video accuracy (mean evidence cost; App. C.11, Table 15):
| Selection rule | Qwen2.5-VL, packets | LLaVA-Video, packets |
|---|---|---|
| Temporal truncation | 63.41 / 53.62 (464) | 65.58 / 56.37 (461) |
| Relevance-only | 64.38 / 54.71 (462) | 67.12 / 57.62 (464) |
| Cost-aware relevance | 63.97 / 54.18 (446) | 66.31 / 56.64 (443) |
| Gain-only greedy (no cost normalization) | 62.89 / 52.47 (454) | 64.97 / 54.58 (455) |
| RCES | 65.33 / 54.78 (467) | 68.30 / 59.22 (450) |
With frame packets, RCES has the highest measured overall and long-video accuracy on both backbones, at comparable evidence cost: its overall gains are 0.95–2.44 points on Qwen2.5-VL and 1.18–3.33 points on LLaVA-Video.
W4: Newer Backbones
We completed RCES evaluations on both newer backbones and a TV-RAG comparison on InternVL3.5-8B (VideoMME, , 64 frames, same budget limits and cost definition). Latency is per-question latency, excluding shared extraction (App. C.12, Table 16):
| Backbone | Method | Accuracy | Short | Medium | Long | Evidence cost | Latency (s/question) |
|---|---|---|---|---|---|---|---|
| Qwen3.5-9B | RCES | 72.96 | 82.89 | 73.56 | 62.44 | 453.6 | 53.7 |
| InternVL3.5-8B | RCES | 68.74 | 79.22 | 69.56 | 57.44 | 469.2 | 20.4 |
| InternVL3.5-8B | TV-RAG | 69.33 | 80.22 | 69.22 | 58.56 | 482.0 | 85.4 |
On InternVL3.5-8B, the measured accuracy difference between RCES and TV-RAG is not statistically significant (68.74 vs 69.33, paired McNemar ), while RCES uses slightly less evidence (469 vs 482 units) and is about four times faster per question (20.4 vs 85.4 s). Due to limited compute during the rebuttal period, the remaining baseline runs on these backbones (TV-RAG and Video-RAG on Qwen3.5-9B, and Video-RAG on InternVL3.5-8B) are still in progress and will be added once complete.
Part 1
Status of additional experiments: TV-RAG at BT=1000 and the remaining newer-backbone baselines (W4) are still in progress. Completed results are incorporated in the prepared revised manuscript. We will upload the revised PDF once these pending runs are complete. Completed results follow.
W1: Broader Validation of the Selector
We thank the reviewer for requesting additional validation. We conducted budget-binding analyses and paired statistical tests to isolate the contribution of the RCES selector.
Uniform-frame evaluation. The binding-pool analysis uses the controlled setup of Table 3: VideoMME, Qwen2.5-VL, 64 uniform frames, BT=800, and identical candidate pools across selection rules.
Budget-binding questions. Although 88.6% of MLVU pools fit within the budget, 1,222 of 2,700 VideoMME questions (45%) require evidence selection, including approximately 88% of long-video questions.
Overall results are in Table 3; binding gains are in Sec. 5.3 and App. C.10, Table 14. RCES outperforms every alternative on binding questions, with gains of 2.72–4.73 percentage points.
Statistical significance. We additionally computed paired 95% bootstrap confidence intervals and exact McNemar tests (App. C.10, Table 14).
All reported confidence intervals exclude zero. The gains are approximately twice as large on binding questions, and the additional correct predictions on these questions equal the overall improvements. This directly supports the effectiveness of RCES when the evidence budget requires an actual selection decision.
Final configuration and second backbone. We extend this comparison to frame packets and to LLaVA-Video in Part 3 (W1, continued).
W2: Attributing the Latency Advantage
Timing boundaries and caching. All methods were run on the same GPU type and memory configuration and timed with the same per-question protocol: latency covers each method's question-dependent processing through the final VLM call, excluding shared pre-extraction. RCES and Video-RAG extract ASR and OCR once per video and cache them. TV-RAG, run with its original code, caches ASR per video but recomputes OCR on all 64 frames, frame decoding, and Contriever document embeddings for each question. For RCES, the one-time extraction (Whisper-large-v3 and EasyOCR) takes 41.6 s per video, about 13.9 s per question when shared by a video's three questions, and is reported separately (App. C.7). Even including this extraction amortized across a video's three questions, RCES (17.3+13.9≈31.2 s) remains faster than TV-RAG (42.4 s). These are therefore comparisons of each method's native pipeline, not measurements that isolate evidence selection.
Video-RAG on LLaVA-Video (43.3 vs 28.7 s). The recorded difference lies almost entirely in preprocessing (frame decoding and CLIP scoring), which precedes text selection:
Retrieval/selection and the final VLM call each differ by less than 0.4 s. The recorded gap is concentrated in preprocessing; the cause of this difference between the separate runs was not isolated.
Stage breakdowns. On Qwen2.5-VL (VideoMME, BT=800, same hardware; caching as described above):
Note: Dashes indicate stages not timed separately. TV-RAG's frame decoding is included in extraction (11.82 s); Video-RAG's ASR/OCR cache loading is included in preprocessing (21.26 s). They do not indicate zero time.