acceptodds
Under review as a conference paper at ICLR 2027

Budgeted Multimodal Evidence Selection for Long-Video Understanding via Knapsack Optimization

Abstract

Understanding long videos with vision–language models (VLMs) requires selecting useful information within limits on the frames and text a model can process. Existing approaches address this by leveraging retrieval augmentation to supplement sampled frames with relevant information, such as speech transcripts and on-screen text, and select the most related evidence items based on similarity thresholds and ranking cutoffs. However, these heuristics do not directly optimize the joint usefulness of heterogeneous, unequal-cost evidence under a fixed model input budget. We make the first attempt in long-video understanding to formulate the final selection of unequal-cost textual evidence as a knapsack-constrained optimization problem, and introduce RCES, a training-free framework that allocates text and frames under separate input budgets without modifying the answering VLM. Specifically, RCES optimizes a monotone submodular objective that balances question relevance, temporal and source coverage, and representativeness of the selected evidence under a knapsack constraint, which can be solved by a density-greedy algorithm with performance guarantees on the objective value. A separate frame selector is further proposed to combine uniformly spaced frames for global context with short frame sequences around question-relevant moments. Across three long-video benchmarks and five VLM backbones, RCES achieves competitive accuracy with substantial reductions in textual evidence cost or latency, compared to related baselines.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
long-video understanding, multimodal large language models, retrieval-augmented generation, budgeted evidence selection

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_6Z6BOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces RCES, a training-free framework for selecting evidence for long-video question answering. Its main contribution is to formulate the selection of unequal-length textual evidence as knapsack-constrained optimization. The objective combines question relevance, temporal coverage, source coverage, and semantic representativeness, and is optimized using density greedy with a best-singleton safeguard. Video frames are selected separately through uniform anchors and question-relevant frame packets. Experiments across three benchmarks and five frozen VLM backbones show competitive accuracy with reductions in evidence volume or latency, depending on the setting. Controlled ablations examine the contributions of the text objective and frame allocator.

Critical strengths
  1. A useful formulation of a practical evidence-selection problem. The paper clearly distinguishes retrieving potentially relevant evidence from deciding what should actually occupy the final prompt. Accounting for unequal text lengths and the marginal value of an evidence set is more appropriate than controlling only the number of retrieved items. The resulting selector requires neither model training nor additional calls to the answering VLM, making it a practical addition to retrieval-based video understanding pipelines.
  2. Meaningful accuracy–efficiency trade-offs with useful controlled ablations. Under the common text budget on VideoMME, RCES uses approximately 24% less textual evidence than TV-RAG, while achieving slightly higher measured accuracy and substantially lower reported latency across three backbones. Importantly, the paper also includes fixed-pool, fixed-frame comparisons and individual objective-term removals, rather than relying only on comparisons against models without retrieved text. These experiments provide concrete support for the proposed selection strategy.
Critical weaknesses
  1. The text selector’s independent benefit needs broader validation. The controlled text-selection comparisons use only Qwen2.5-VL on VideoMME at BT=800B_T=800 with uniform frames, leaving unclear whether the gains persist in the final packet-based pipeline or other settings. Moreover, 88.6% of the distilled MLVU pools already fit within the budget, so most examples do not require the selection trade-offs central to the paper. Please extend the fixed-pool comparisons to the final frame configuration and at least one additional benchmark or backbone, separately report examples whose candidate pools exceed the budget, and provide paired confidence intervals for the selector’s gains.

  2. The reported latency advantage is difficult to attribute to evidence allocation. On LLaVA-Video/VideoMME, Video-RAG takes 43.3 seconds with 472 evidence units under the common cap, but only 28.7 seconds with its larger native input of 1,623 units. This unexplained pattern makes it difficult to distinguish savings from evidence allocation from differences in preprocessing, caching, or implementation. Please clarify the timing boundaries and caching protocol across methods, explain this discrepancy, and provide comparable breakdowns for evidence extraction, retrieval/selection, and final VLM inference under matched hardware and cache conditions.

  3. The accuracy–cost evaluation relies too heavily on a single operating point. The main comparisons use either a common 800-unit cap or competitors’ native text footprints, while the budget sweep evaluates only RCES. These results do not establish how the methods compare across resource limits. Furthermore, the character-based cost measure is a proxy: its reported correlation with tokenizer counts does not establish equivalent actual input costs across methods. Please specify how each baseline is capped and compare RCES with the strongest retrieval baselines across several budgets, reporting accuracy against measured input tokens and end-to-end latency with the same backbone and frame budget. This would establish whether the advantage extends beyond the selected 800-unit setting.

  4. Generalization to newer VLMs remains untested. The evaluation is limited to earlier-generation backbones, including Qwen2.5-VL, LLaVA-Video, and InternVL2.5. This leaves unclear whether RCES’s accuracy–efficiency gains persist on newer models: the benefit of auxiliary text and the savings from input reduction may change with the backbone’s multimodal capabilities and inference architecture. Please compare RCES with the strongest retrieval baselines on recent, comparably sized models, such as Qwen3.5-9B and InternVL3.5-8B.

Questions for authors

see weakness.

6 comments

Status of additional experiments: TV-RAG at BT=1000B_T = 1000 and the remaining newer-backbone baselines (W4) are still in progress. Completed results are incorporated in the prepared revised manuscript. We will upload the revised PDF once these pending runs are complete. Completed results follow.

W1: Broader Validation of the Selector

We thank the reviewer for requesting additional validation. We conducted budget-binding analyses and paired statistical tests to isolate the contribution of the RCES selector.

Uniform-frame evaluation. The binding-pool analysis uses the controlled setup of Table 3: VideoMME, Qwen2.5-VL, 64 uniform frames, BT=800B_T = 800, and identical candidate pools across selection rules.

Budget-binding questions. Although 88.6% of MLVU pools fit within the budget, 1,222 of 2,700 VideoMME questions (45%) require evidence selection, including approximately 88% of long-video questions.

Selection ruleAll questions (%)Binding only (%)
Temporal truncation61.9354.40
Relevance-only62.5255.73
Cost-aware relevance62.2255.07
Gain-only greedy (no cost normalization)61.6053.72
RCES63.7458.45

Overall results are in Table 3; binding gains are in Sec. 5.3 and App. C.10, Table 14. RCES outperforms every alternative on binding questions, with gains of 2.72–4.73 percentage points.

Statistical significance. We additionally computed paired 95% bootstrap confidence intervals and exact McNemar tests (App. C.10, Table 14).

RCES vs.Overall gainOverall 95% CIp-valueBinding gainBinding 95% CI
Temporal truncation+1.81[0.75, 2.87]0.001+4.05[1.68, 6.42]
Relevance-only+1.22[0.30, 2.14]0.010+2.72[0.56, 4.88]
Cost-aware relevance+1.52[0.52, 2.52]0.003+3.38[1.15, 5.61]
Gain-only greedy (no cost normalization)+2.14[0.95, 3.33]<0.001+4.73[2.18, 7.28]

All reported confidence intervals exclude zero. The gains are approximately twice as large on binding questions, and the additional correct predictions on these questions equal the overall improvements. This directly supports the effectiveness of RCES when the evidence budget requires an actual selection decision.

Final configuration and second backbone. We extend this comparison to frame packets and to LLaVA-Video in Part 3 (W1, continued).

W2: Attributing the Latency Advantage

Timing boundaries and caching. All methods were run on the same GPU type and memory configuration and timed with the same per-question protocol: latency covers each method's question-dependent processing through the final VLM call, excluding shared pre-extraction. RCES and Video-RAG extract ASR and OCR once per video and cache them. TV-RAG, run with its original code, caches ASR per video but recomputes OCR on all 64 frames, frame decoding, and Contriever document embeddings for each question. For RCES, the one-time extraction (Whisper-large-v3 and EasyOCR) takes 41.6 s per video, about 13.9 s per question when shared by a video's three questions, and is reported separately (App. C.7). Even including this extraction amortized across a video's three questions, RCES (17.3+13.9≈31.217.3+13.9\approx31.2 s) remains faster than TV-RAG (42.4 s). These are therefore comparisons of each method's native pipeline, not measurements that isolate evidence selection.

Video-RAG on LLaVA-Video (43.3 vs 28.7 s). The recorded difference lies almost entirely in preprocessing (frame decoding and CLIP scoring), which precedes text selection:

Stage (s)Capped (472 units)Native (1,623 units)
Preprocessing36.5122.42
Retrieval and selection2.602.54
Final VLM call4.163.78
Total43.2728.73

Retrieval/selection and the final VLM call each differ by less than 0.4 s. The recorded gap is concentrated in preprocessing; the cause of this difference between the separate runs was not isolated.

Stage breakdowns. On Qwen2.5-VL (VideoMME, BT=800B_T = 800, same hardware; caching as described above):

Stage (s)RCESTV-RAGVideo-RAG
Evidence extraction / cache read0.05 (included below)11.82—
Frame preprocessing / scoring5.79—21.26
Retrieval and selection1.295.792.34
Final VLM call10.1931.581.54
End-to-end latency (Table 1)17.2742.3525.14

Note: Dashes indicate stages not timed separately. TV-RAG's frame decoding is included in extraction (11.82 s); Video-RAG's ASR/OCR cache loading is included in preprocessing (21.26 s). They do not indicate zero time.

TV-RAG has higher measured extraction/cache-access time (11.82 vs 0.05 s), retrieval/selection time (5.79 vs 1.29 s), and final VLM inference time (31.58 vs 10.19 s). RCES cache access is included in its retrieval/selection time. These stage measurements are not additive. Against Video-RAG, which caches ASR and OCR like RCES, the situation differs: RCES is faster overall because of its lighter frame preprocessing (5.8 vs 21.3 s), while Video-RAG's selection and final VLM call are shorter. We therefore do not attribute the latency gap to Video-RAG to evidence allocation, and we note this as a limitation of the latency comparison. TV-RAG's stage timers are recorded separately; their sum (49.2 s) exceeds its end-to-end latency.

Where the time goes. Per question, RCES does three simple steps: it scores the frames with CLIP (5.79 s), reads the cached speech and on-screen text and selects from it (1.29 s in total, of which the cache read takes 0.05 s and the knapsack selection 0.002 s), and makes one final VLM call (10.19 s), for 17.27 s in total (App. C.7, Table 11). The latency advantage therefore comes from this pipeline design (cached evidence, lightweight selection, and a single VLM call), not from the knapsack optimization itself, which accounts for a negligible share of the time.

W3: Comparison Across Operating Points

How each baseline is capped. All retrieval methods are limited by the same text budget BTB_T, measured with the same cost proxy as RCES (cT(e)=max⁡(1,⌊∣text(e)∣/4⌋)c_T(e)=\max(1,\lfloor\lvert\mathrm{text}(e)\rvert/4\rfloor), Eq. 1). Each baseline keeps its own retrieval, ranking, and preprocessing unchanged; only its final evidence is limited. Baselines add their retrieved items in their native order: Video-RAG stops at the first item that does not fit, and TV-RAG cuts its concatenated evidence at the budget, so its last item may be truncated. RCES instead skips items that do not fit and continues selecting. Video-RAG nevertheless uses a comparable mean evidence cost: under BT=800B_T = 800 it uses 451 units versus 467 for RCES (Table 1). The backbone, frame count (BV=64B_V = 64), hardware, and text budget are matched across methods.

Baselines across budgets. We evaluated the strongest retrieval baselines at additional budgets on VideoMME (Qwen2.5-VL-7B, 64 frames, same budget limits and cost definition):

MethodBT=500B_T = 500BT=800B_T = 800BT=1000B_T = 1000Paired test between budgets
Video-RAG62.1561.0064.41500 vs 800: p=0.15p = 0.15 (n.s.); 800 vs 1000: p<0.001p < 0.001
TV-RAG63.6364.04pending500 vs 800: p=0.50p = 0.50 (n.s.)
RCES64.1965.3365.00500 vs 800: p=0.013p = 0.013; 800 vs 1000: p=0.38p = 0.38 (n.s.)

Among the completed runs, RCES has the highest measured accuracy at every budget (App. C.4, Table 10). Video-RAG's difference between 500 and 800 is not significant, and it improves only at BT=1000B_T = 1000, where RCES remains higher (65.00 vs 64.41). The TV-RAG run at BT=1000B_T = 1000 is still in progress and will be added once complete.

Accuracy against measured input tokens. To check the cost proxy against actual textual evidence tokens, we counted the evidence tokens sent to the model with the Qwen2.5-VL tokenizer, applied identically to every method (App. A.2, Table 4):

MethodBTB_TAccuracy (%)Mean evidence tokensCost units (Eq. 1)
RCES50064.19347319
Video-RAG50062.15370321
TV-RAG50063.63362347
RCES100065.00566554
Video-RAG100064.41594558

At these operating points, the mean token counts are close to the cost units for every method (1.0–1.2 tokens per unit), so the methods receive comparable evidence volumes under the common budget. At each budget, RCES has the highest measured accuracy with the fewest mean evidence tokens.

Latency across budgets. Per-question latency, shared extraction excluded (same backbone, frames, and hardware; seconds per question; App. C.4, Table 10):

MethodBT=500B_T = 500BT=800B_T = 800BT=1000B_T = 1000
Video-RAG30.425.130.3
TV-RAG19.342.4pending
RCES14.317.318.6

Among the completed runs, RCES is the fastest at every budget, and its latency grows only moderately with the budget (14.3 to 18.6 s). Baseline timings were obtained in separate runs with their native preprocessing and caching policies; these measurements do not isolate the effect of changing the text budget. The TV-RAG run at BT=1000B_T = 1000 is still in progress and will be added once complete.

W1 (continued): Final Frame Configuration and Second Backbone

We extended the fixed-pool comparison to the final configuration with frame packets (16 anchors + 12×4 = 64 frames), on Qwen2.5-VL and on a second backbone, LLaVA-Video-7B (VideoMME, BT=800B_T = 800, identical candidate pools). Entries are overall / long-video accuracy (mean evidence cost; App. C.11, Table 15):

Selection ruleQwen2.5-VL, packetsLLaVA-Video, packets
Temporal truncation63.41 / 53.62 (464)65.58 / 56.37 (461)
Relevance-only64.38 / 54.71 (462)67.12 / 57.62 (464)
Cost-aware relevance63.97 / 54.18 (446)66.31 / 56.64 (443)
Gain-only greedy (no cost normalization)62.89 / 52.47 (454)64.97 / 54.58 (455)
RCES65.33 / 54.78 (467)68.30 / 59.22 (450)

With frame packets, RCES has the highest measured overall and long-video accuracy on both backbones, at comparable evidence cost: its overall gains are 0.95–2.44 points on Qwen2.5-VL and 1.18–3.33 points on LLaVA-Video.

W4: Newer Backbones

We completed RCES evaluations on both newer backbones and a TV-RAG comparison on InternVL3.5-8B (VideoMME, BT=800B_T = 800, 64 frames, same budget limits and cost definition). Latency is per-question latency, excluding shared extraction (App. C.12, Table 16):

BackboneMethodAccuracyShortMediumLongEvidence costLatency (s/question)
Qwen3.5-9BRCES72.9682.8973.5662.44453.653.7
InternVL3.5-8BRCES68.7479.2269.5657.44469.220.4
InternVL3.5-8BTV-RAG69.3380.2269.2258.56482.085.4

On InternVL3.5-8B, the measured accuracy difference between RCES and TV-RAG is not statistically significant (68.74 vs 69.33, paired McNemar p=0.48p = 0.48), while RCES uses slightly less evidence (469 vs 482 units) and is about four times faster per question (20.4 vs 85.4 s). Due to limited compute during the rebuttal period, the remaining baseline runs on these backbones (TV-RAG and Video-RAG on Qwen3.5-9B, and Video-RAG on InternVL3.5-8B) are still in progress and will be added once complete.

Reviewer_b5xgOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes to formulate the choice of textual evidence that goes into the prompt for long-video question answering as monotone submodular maximization under a knapsack constraint. The candidates are retrieved, time-stamped entries from speech transcripts, on-screen text, and object detection. Their lengths differ, and cost is measured in characters. The objective is a weighted sum of query relevance, temporal coverage, source coverage, and representativeness, solved by cost-scaled greedy with a single-element fallback [4], with a constant-factor guarantee on this surrogate objective. Frames are allocated separately, partly uniform and partly around question-relevant moments, and everything is passed to a frozen VLM in one call. Overall, the positioning of this paper aligns with the evidence, but some claims are not sufficiently supported. I will finalize my score according to the authors' rebuttal.

Critical strengths

The problem and motivation are well posed. The paper separates the evidence that is retrieved from the evidence that actually reaches the model, and points out that because entries differ in length, the budget should limit total length rather than the number of entries. Existing retrieval-based long-video methods only handle this indirectly through similarity thresholds and truncation. I appreciate this motivation.

The theory is presented honestly. The objective is shown to be monotone submodular, the paper is clear that the guarantee covers the surrogate objective and not QA accuracy, it states that a lighter greedy variant is used in practice, and the appendix compares greedy against the exact solution.

The efficiency gains are clear and consistent. At equal text budgets on three backbones, latency is lower than every retrieval baseline, and against TV-RAG both text and latency drop substantially at nearly the same accuracy (Tables 1 and 2). This is the most solid result in the paper.

Critical weaknesses
  1. The budget does not bind on most samples, so knapsack selection usually takes everything. The appendix shows that most cleaned pools already fit in the budget and that actual usage stays well below the cap. When the pool fits, greedy selects every candidate and makes no real choice, which likely also explains why greedy almost always matches the exact optimum. The core claim is that cost-aware selection under a budget beats heuristic retrieval, but if the budget rarely binds, most of the main experiments never really test the selection.

  2. The paper's own ablation shows that accounting for cost does not help. In Table 3, cost-aware ranking is slightly worse than relevance-only ranking. The small gain of the full objective comes from the diversity terms, which can be added to plain top-k without a knapsack (Lin & Bilmes [1]; MDP3 [2] and FLoC [3] on the visual side), and there is no comparison with MMR-style selection. Add the first point, and the fact that accuracy is on par with TV-RAG on every benchmark (without a cap, TV-RAG is even more accurate on VideoMME), and there is no evidence yet that knapsack selection improves accuracy, which is the point of the formulation.

References

[1] Lin & Bilmes. Multi-document summarization via budgeted maximization of submodular functions. NAACL 2010.

[2] Sun et al. MDP3: A training-free approach for list-wise frame selection in video-LLMs. ICCV 2025;

[3] Cho et al. FLoC: Facility location-based efficient visual token compression for long video understanding. arXiv:2511.00141, 2025.

Questions for authors
  1. Among the 220 questions in App. C.8 where greedy reaches the exact optimum, how many have pools that fit entirely in the budget?
  2. Could you compare the selection rules only on samples where the budget binds?
  3. How does an MMR-style top-k baseline, using the same diversity terms without the knapsack, compare?
  4. What are the pool sizes before and after cleaning? Cleaning likely explains why the pools fit the budget, and it may drop details needed to answer. A control with cleaning followed by simple top-k would separate the two.
  5. Were the retrieval baselines run in their native settings, for example Video-RAG with detection text on? Detection is disabled in the main RCES configuration.
  6. The latency breakdown shows that most of the time goes to CLIP preprocessing, while selection and the final VLM call take a small share. What then explains the latency gap to TV-RAG? And since Video-RAG uses about as much text as RCES under the same cap (Table 1), is the large text reduction mainly due to comparing against uncapped baselines?
5 comments

We thank the reviewer for the constructive assessment. Completed analyses are incorporated in the prepared revised manuscript. We will upload the revised PDF once the pending baseline runs are complete.

Q1: Exact Optimization on Budget-Binding Questions

Of the 220 instances evaluated in App. C.8 of the submission, 190 (86.4%) fit within BT=800B_T = 800, making greedy trivially optimal. We therefore additionally examined all 197 binding questions of the 421-question set used for this analysis (including the 30 among the 220).

For computationally challenging pools, exact optimization was restricted to the 18 most relevant candidates, and greedy was evaluated on the identical subset to ensure a fair comparison.

Binding-pool evaluationQuestionsMean greedy/exactExactly optimalWithin 1%Minimum ratio
Full candidate pool300.995986.7%96.7%0.8805
Top-18 restricted pool1670.997855.1%97.6%0.9415
Combined1970.997559.9%97.5%0.8805

On binding instances, greedy achieves 99.75% of the corresponding exact objective value on average, with 97.5% of solutions within 1% of the optimum. For the 167 restricted cases, optimality refers specifically to the 18-item subproblem.

These results clarify that the original 98.2% exact-optimality rate is influenced by non-binding pools, while the revised analysis demonstrates strong optimization quality when selection is required. The prepared revision includes this binding-only analysis (App. C.9, Table 13).

Q2: Selection Performance When the Budget Binds

On VideoMME, approximately 45% of questions have cleaned evidence pools exceeding BT=800B_T = 800. We therefore re-evaluated Table 3's selection rules specifically on these binding questions, keeping the candidate pool, VLM, 64 uniform frames, and budget identical.

Selection ruleOverall (%)Binding-only (%)
Temporal truncation61.9354.40
Relevance-only62.5255.73
Cost-aware relevance62.2255.07
RCES63.7458.45

RCES outperforms the alternatives by 2.72–4.05 percentage points on binding questions. The additional correct predictions on binding questions (49, 33, and 41, respectively) equal the overall improvements, indicating that the accuracy gains arise entirely from instances requiring evidence selection rather than those where the entire pool fits. Overall results appear in Table 3; binding-only results are reported in Sec. 5.3.

Q3: Diversity-Aware Selection Without Cost Normalization

We added a diversity-aware gain-only greedy baseline using the same four RCES objective terms (relevance, temporal coverage, source coverage, and representativeness) without cost normalization.

At each step, this baseline selects the feasible item with the largest marginal objective gain, without dividing by evidence cost. RCES instead selects according to marginal gain per cost. Both respect BT=800B_T = 800 and use identical candidates, frames, and VLM settings.

Selection ruleOverall (%)Long (%)Binding-only (%)Mean cost
Gain-only greedy (no cost normalization)61.6051.2053.72453
RCES (cost-normalized greedy)63.7454.4458.45467

RCES improves overall accuracy by 2.14 points and long-video accuracy by 3.24 points while consuming only 14 additional cost units (3.1%). On binding questions, where a choice is required, the gain rises to 4.73 points (paired 95% CI [2.18, 7.28]).

Because both methods use the same objective terms, this comparison provides direct evidence for the benefit of cost-normalized selection beyond diversity-aware marginal-gain selection alone. The prepared revision includes this baseline (Table 3; Sec. 5.3; binding gains in App. C.10, Table 14).

Q4: Cleaning, Pool Sizes, and Top-k Controls

Pool sizes. On VideoMME, cleaning reduces mean candidate-pool cost from 2,176 to 1,273 units (median: 794 to 606). Near-duplicate merging accounts for 41.3% of raw evidence cost, while filler removal accounts for only 0.1%.

Budget binding. The proportion of pools exceeding BT=800B_T = 800 decreases modestly, from 49.9% before cleaning to 45.2% afterward. Among long videos, the corresponding proportions are 91.9% and 88.2%. Thus, cleaning changes the binding status of only about 125 of 2,700 questions.

Cleaning control. Running RCES without cleaning on all 2,700 questions gives 64.85% versus 65.33% with cleaning (−0.48, p=0.40p = 0.40), with no statistically detectable difference on any split: short +0.11 (p=1.0p = 1.0), medium −0.33 (p=0.77p = 0.77), and long −1.22 (p=0.38p = 0.38). We did not detect a statistically significant accuracy difference with and without cleaning.

Cleaning + simple selection. Table 3 already compares relevance-only ranking and temporal truncation against RCES using identical cleaned pools. Their overall accuracies are 62.52% and 61.93%, versus 63.74% for RCES. On binding questions, they achieve 55.73% and 54.40%, versus 58.45% for RCES.

These controls distinguish the effects of cleaning from those of the final selection rule.

Q5: Native Baseline Configurations

All retrieval baselines were reproduced using their original implementations and method-specific components, including APE object detection where applicable. Their core retrieval logic was preserved, with the common text-budget constraint applied for matched-budget comparisons.

RCES operates without detection, as stated in Sec. 5.1. Consequently, the baselines retain access to detection evidence where their original methods use it, while RCES achieves competitive accuracy without that source.

Q6: Latency and Evidence-Cost Comparisons

Latency relative to TV-RAG. We measured the following components on VideoMME using Qwen2.5-VL-7B and BT=800B_T = 800:

ComponentTV-RAG (s)RCES (s)
Evidence extraction/cache access11.820.05 (included in retrieval/selection)
Retrieval/selection, including planning5.791.29
CLIP frame scoring—5.79
Final VLM inference31.5810.19
Reported latency42.3517.27

The dash denotes a stage not timed separately. The component timers are measured separately and are not strictly additive; TV-RAG's individual measurements sum to 49.2 s, exceeding its reported end-to-end latency.

The largest measured differences occur in final VLM inference (31.58 versus 10.19 s) and evidence preparation. RCES reuses cached evidence, and its filtering, clustering, and knapsack selection require approximately 0.002 s/question. These latencies compare each method's native pipeline rather than isolating evidence selection.

Native versus matched text consumption. The largest evidence-cost reductions arise in native-footprint comparisons (Table 2), where retrieval baselines retain their usual evidence volume.

Under the common BT=800B_T = 800 cap (Table 1), the relevant comparison is accuracy and latency at controlled evidence budgets:

  • Video-RAG: RCES achieves 65.33% versus 61.00% accuracy, with comparable evidence cost (467 versus 451) and lower latency (17.3 versus 25.1 s).
  • TV-RAG: RCES achieves 65.33% versus 64.04% accuracy, with lower evidence cost (467 versus 616) and latency (17.3 versus 42.4 s).

We distinguish these evaluation settings explicitly to avoid conflating native-footprint efficiency with matched-budget improvements.

W2: Cost-Aware Relevance Versus Full RCES

Cost-aware relevance divides individual relevance scores by evidence cost but does not account for temporal coverage, source coverage, or representativeness. It therefore favors short, relevant snippets without explicitly considering their collective utility.

This may help explain why cost-aware relevance alone does not outperform relevance-only ranking (62.22% versus 62.52%; Table 3).

The additional experiments show the benefit of joint cost-aware selection:

  • Using identical objective terms, RCES improves over gain-only greedy by 2.14 percentage points (Q3).
  • On binding questions, RCES improves over cost-aware relevance by 3.38 points (58.45% versus 55.07%; Q2).

Thus, the results support combining evidence cost with set-level coverage and representativeness, rather than applying cost normalization to relevance alone.

W2: Accuracy Relative to TV-RAG

We acknowledge that native-footprint TV-RAG achieves higher VideoMME accuracy while using substantially more textual evidence (mean cost 1,481 versus 450 for RCES in Table 2).

Our goal is competitive accuracy with a substantially smaller evidence footprint and lower measured latency.

Under identical candidate pools, visual inputs, and text budgets, RCES improves over simpler selection rules, with the clearest gains on budget-binding questions (Q2, Q3).

The prepared revision includes the binding-question analyses, the gain-only greedy baseline, and the extended greedy-versus-exact comparison.

We have uploaded the revised manuscript, including budget-binding selector evaluations, the gain-only greedy control, and the expanded greedy-versus-exact analysis (Sec. 5.3; Apps. C.9-C.10). The updated PDF is available on the submission page. We welcome any follow-up questions before the discussion deadline.

Reviewer_8A1POpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes RCES, a training-free framework for budgeted evidence selection in long video understanding. RCES uses text information from OCR/ASR along with the visual frames to provide the answer. The paper formulates final textual evidence selection as a budgeted knapsack optimization problem. The paper evaluates on VideoMME, MLVU and LongVideobench over baseline uniform sampling, ASK, FOCUS and other methods as reported in Table 1.

Overall, even though the motivation is unambiguous, the algorithmic novelty of the paper is limited, as RCES uses existing approaches and repositions them well in this paper. The paper needs more ablation studies on the budget allocation for OCR/ASR extraction and baseline comparisons with existing methods on the same visual budget.

Critical strengths

A simple training-free approach for log video understanding. The paper first reports the unequal-cost textual evidence as a knapsack-constrained optimization problem for long-video understanding. The budget allocation between textual (OCR/ASR) and visual information is intuitive and helps to improve the overall accuracy.

Critical weaknesses

RCES is a well-formulated approach for long video understanding; it uses visual + textual evidence for long video question answering but lacks overall algorithmic novelty. The paper needs more ablation, i.e the end-to-end latency is not clearly reported in the paper, need more details about the OCR+ASR extraction latency, does these latency has been included in the reported latency in table 1, also lacks the comparable bencmarks with eual frame budget used in some of the rencet methods i.e FOCUS [1]

[1] Zirui Zhu et al. FOCUS: Efficient keyframe selection for long video understanding. In International Conference on Learning Representations, 2026. URL: https://arxiv.org/pdf/2510.27280

Questions for authors
  1. RCES uses which model for ASR/OCR extrcation, does these information pre-extracted, what is the latency for these models and does the table 1 reported latency includes latency for OCR and ASR extraction? Also, for near-duplicate filtering of the text, how much latency budget is allocated for that?

  2. RCES uses BT = 800, but how was the BT cap of 800 characters decided? Can the author provide more ablation for different BT budgets?

  3. The reported performance in Table 1: RCES uses BV=64 (64 visual frames). FOCUS[1] reported performance with a 32-frame budget, so the comparison in Table 1 is not aligned with an equal frame budget. Can the author provide the RCES performance on a BV=32 frame budget? In Table 1, is the reported performance of FOCUS[1] based on 32 frames or reproduced for 64 frames?

[1] Zirui Zhu et al. FOCUS: Efficient keyframe selection for long video understanding. In International Conference on Learning Representations, 2026. URL: https://arxiv.org/pdf/2510.27280

4 comments

We thank the reviewer for the constructive assessment. Completed analyses are incorporated in the prepared revised manuscript. We will upload the revised PDF once the pending baseline runs are complete.

W1: Algorithmic Novelty

Our contribution is the formulation of final, post-retrieval ASR/OCR evidence selection under an explicit unequal-cost budget. RCES uses an established density-greedy solver with a best-singleton safeguard (Sec. 4, Alg. 1); we do not claim a new optimization algorithm.

Unlike thresholding, ranking, or truncation alone, RCES optimizes the selected evidence set under a shared budget. Its objective combines relevance, temporal coverage, source coverage, and representativeness. Monotone submodularity provides an approximation guarantee on the objective value, not QA accuracy. The selector is training-free and requires no iterative VLM calls.

W2 / Q1: Extraction Models and Latency

  • Models: Whisper-large-v3 for ASR and EasyOCR for OCR (App. A.1; now Sec. 5.1). Contriever retrieves text and CLIP ViT-L/14-336 scores frames (App. A.1).
  • RCES pre-extraction: ASR/OCR are extracted once per video and cached across VideoMME's 3 questions (cache loading: ≈0.05\approx 0.05 s/question).
  • Timing protocol: All Table 1 methods use the same per-question timing boundary, excluding shared pre-extraction. RCES and Video-RAG read cached ASR/OCR; TV-RAG, run with its original code, caches ASR and recomputes OCR per question. Even counting RCES's one-time extraction amortized across a video's three questions, RCES (≈31.2\approx 31.2 s) remains faster than TV-RAG (42.4 s).
  • Reported latency: RCES requires 17.27 s/question on Qwen2.5-VL: 5.79 s frame scoring, 1.29 s retrieval/selection, and 10.19 s VLM inference (App. C.7, Table 11).
  • Extraction overhead: 7.1/22.9/94.8 s per short/medium/long video, averaging 41.6 s/video (≈13.9\approx 13.9 s/question amortized).
  • Selection overhead: Filtering, medoid clustering, and knapsack selection together take ≈0.002\approx 0.002 s/question.

Sec. 5.1 now distinguishes shared extraction costs from the post-extraction latency used for Table 1 comparisons.

Q2: Justification of BT=800B_T = 800

Text cost uses tokenizer-independent units, cT(e)=max⁡(1,⌊characters(e)/4⌋)c_T(e)=\max(1,\lfloor\mathrm{characters}(e)/4\rfloor), which correlate with tokenizer counts (r=0.95r = 0.95; App. A.2).

We selected BT=800B_T = 800 using VideoMME and kept it fixed across the remaining datasets and backbones. Extending the controlled budget sweep yields (revised Table 3):

BTB_TOverall acc. (%)Long-video acc. (%)Mean cost
30062.4051.10252
50064.1954.22319
80065.3354.78467
100065.0055.67554
120065.1956.30721

Among the tested budgets, 800 achieves the highest measured overall accuracy, significantly exceeding 300 (p=0.0062p = 0.0062) and 500 (p=0.013p = 0.013); both remain significant after Holm correction.

Larger budgets increase mean evidence cost by 18.6% (1000) and 54.4% (1200), without statistically significant overall accuracy differences (p=0.38p = 0.38 and 0.46).

Long-video accuracy increases with larger budgets (54.78% at 800 to 56.30% at 1200). Thus, BT=800B_T = 800 offers a favorable overall accuracy–cost trade-off rather than a universally optimal budget.

W3 / Q3: Frame-Budget Fairness and FOCUS

All Table 1 methods, including FOCUS, were reproduced by us under the same BV=64B_V = 64 frame budget. We did not reuse FOCUS's published 32-frame results.

The reproduced FOCUS accuracies are 61.11%, 61.11%, and 60.33% on Qwen2.5-VL, LLaVA-Video, and InternVideo2.5, respectively.

With Qwen2.5-VL and BT=800B_T = 800, RCES achieves 64.30% at BV=32B_V = 32, compared with 65.33% at BV=64B_V = 64 (reported in the original submission, App. C.5). Even with half the frames, RCES remains above FOCUS at 64 frames (61.11%).

We now explicitly state the common-frame reproduction protocol in Sec. 5.1.

Additional Ablation: ASR/OCR Budget Allocation

RCES imposes no fixed ASR/OCR split. Both sources compete under the shared BT=800B_T = 800 budget according to marginal utility per unit cost.

On VideoMME, RCES ends up allocating approximately 91% of selected evidence cost to ASR and 9% to OCR on average, with OCR retained in 79% of questions.

The observed OCR share varies by question type: 16.2% for spatial perception, 13.7% for object recognition, and 6.0% for temporal reasoning.

Under identical configurations, changing only the available evidence sources yields (Sec. 5.3):

MethodOverall accuracy (%)
RCES (ASR + OCR)65.33
ASR-only63.10 (p=0.007p = 0.007)
OCR-only61.48 (p<0.001p < 0.001)

These results support using both evidence sources; the allocation statistics show that their selected proportions vary across questions. These analyses are included in the prepared revision (Sec. 5.3).

We have uploaded the revised manuscript addressing your comments on the contribution, extraction costs, frame-budget fairness, and budget and source ablations (Sec. 5.3; Table 4). The updated PDF is available on the submission page. We welcome any follow-up questions before the discussion deadline.

Dear authors, thanks for clarifying the details in your responses. At the same budget of 64 frames, RCES improves performance from 61.11% to 65.33% compared to FOCUS using Qwen2.5-VL. So the performance improvement is quite good, but it comes with a significant increase in the latency budget if RCES includes the one-time ASR/OCR extraction latency, averaging 41.6 s/video.

For practical long video understanding applications, both frame and latency budget are quite important to consider. Given the conclusion, I would like to maintain the same rating.

We thank the reviewer for the follow-up and for recognizing the accuracy improvement over FOCUS at the same 64-frame budget. We respectfully disagree with the conclusion that the extraction cost outweighs this improvement, and explain why below.

All requested analyses were completed. In response to the review, we added: (i) the ASR/OCR extraction models and their latency (App. C.7); (ii) a text-budget ablation from BT=300B_T=300 to 12001200 with paired tests (Sec. 5.3); (iii) FOCUS reproduced at the same 64-frame budget, together with RCES at 32 frames (Secs. 5.1 and 5.3); and (iv) an ASR/OCR source-allocation analysis (Sec. 5.3).

Frames-only versus multimodal evidence. FOCUS uses a single channel, video frames. RCES uses three: frames, speech (ASR), and on-screen text (OCR). Combining these channels is standard in practical video understanding systems; for example, assistants on smart glasses such as Ray-Ban Meta process both what the camera sees and what is spoken. OCR makes on-screen text explicitly readable, and ASR adds speech. Removing either source lowers accuracy (ASR-only 63.10%, p = 0.007; OCR-only 61.48%, p < 0.001; Sec. 5.3). With half the frames (BV=32B_V=32), RCES still reaches 64.30%, above FOCUS at 64 frames.

Direct accuracy-latency comparison. Table 1 compares FOCUS and RCES on three backbones with the same 64-frame budget. RCES achieves higher measured accuracy on all three, with competitive per-question latency while incorporating frames, ASR transcripts, and OCR text. The source ablations above support the usefulness of both textual evidence streams.

BackboneFOCUS accuracy (%)FOCUS latency (s)RCES accuracy (%)RCES latency (s)
Qwen2.5-VL-7B61.1115.665.3317.3
LLaVA-Video-7B61.1119.768.3015.2
InternVideo2.5-8B60.338.467.909.7

RCES uses BT=800B_T=800; FOCUS receives no retrieved text. Latency is the reported mean per question; RCES's one-time ASR/OCR extraction cost is additional and reported separately (App. C.7). Bold marks the higher accuracy and lower latency within each backbone.

Extraction latency. We respectfully disagree that the one-time extraction should be counted as per-question latency. ASR/OCR extraction runs once per video, not once per question:

  • The extracted text is reused for every question about that video. On VideoMME's three questions per video, this amortizes to about 14 s per question, and the per-question share decreases as more questions are asked.
  • Extraction runs much faster than real time: 95 s on average for long videos of 30--60 minutes.
  • Once a question is asked, RCES takes 17.3 s versus 15.6 s for FOCUS: 1.7 s more for 4.2 points of accuracy.

Efficiency reporting. We agree that frame and latency budgets both matter in practice. This is why we report the frame and text budgets and per-question latency for each comparison, and separately report RCES's one-time extraction cost (App. C.7).


We believe these results address the reviewer's concerns about the frame and latency budgets, and we hope the reviewer will consider them in the final assessment.

Comments on the paper

4 comments

We thank the reviewers for their constructive feedback. Completed analyses are incorporated in the prepared revised manuscript. We will upload the revised PDF once TV-RAG at BT=1000B_T = 1000 and the remaining newer-backbone baseline runs are complete. The main updates are:

  • Contribution: Clarified that RCES introduces a budgeted textual-evidence selection formulation using an established optimizer. The approximation guarantee concerns the selection objective, not QA accuracy.
  • Selector validation: Added budget-binding evaluations, a gain-only greedy baseline, fixed-pool comparisons with frame packets and a second backbone, paired confidence intervals, and an expanded greedy-versus-exact analysis that distinguishes full-pool and restricted-pool comparisons.
  • Operating points: Extended the text-budget sweep, baseline comparisons, tokenizer-based evidence measurements, and ASR/OCR source analyses.
  • Evaluation protocol: Clarified baseline reproduction, frame budgets, latency boundaries, and caching differences. Additional timing breakdowns are provided in the individual replies.
  • Newer backbones: Added completed evaluations on Qwen3.5-9B and InternVL3.5-8B.

Paper references in the responses refer to the prepared revision. The individual replies present the completed results and address each reviewer’s concerns in detail.

We thank the reviewers and Area Chair for their feedback. The revised manuscript is now available on the submission page. It includes newer-backbone comparisons (Table 2), completed TV-RAG results at BT=1000B_T=1000 (App. C.4), additional selector and budget analyses, and clarifications of reproduction and latency protocols. The individual follow-ups identify the updates relevant to each review. We welcome further questions before the discussion deadline.

We thank the reviewers again for their constructive feedback. We have uploaded the revised manuscript, with changes highlighted in blue. Below, we summarize the updates relevant to each reviewer; detailed responses remain in the individual review threads.

Reviewer 8A1P

  • Clarified that the contribution is the budgeted evidence-selection formulation using an established greedy optimizer (Secs. 1 and 4).
  • Specified the ASR/OCR models, caching, latency boundaries, and separately reported extraction costs (Sec. 5.1; App. C.7).
  • Clarified baseline reproduction under the stated frame budgets and added text-budget and ASR/OCR source analyses (Secs. 5.1 and 5.3; Table 4).

Reviewer b5xg

  • Added the gain-only greedy control and budget-binding selector evaluations with paired confidence intervals (Sec. 5.3; Tables 4 and 15).
  • Expanded the greedy-versus-exact analysis, reporting full-pool and restricted top-18 comparisons separately (App. C.9; Table 14).
  • Clarified centroid-based candidate clustering (Sec. 4.2; App. A.3).

Reviewer 6Z6B

  • Added fixed-pool selector comparisons with frame packets on Qwen2.5-VL and LLaVA-Video (App. C.11; Table 16).
  • Extended budget comparisons and measured-token validation, including the completed TV-RAG result at BT=1000B_T=1000 (Tables 5 and 11).
  • Added newer-backbone comparisons on VideoMME and clarified RCES latency accounting (Tables 2 and 12). Table 2 supersedes the preliminary newer-backbone results and their associated statistical comparison in our earlier reply.

We welcome any follow-up questions or remaining concerns before the discussion deadline.

Dear Area Chair and Reviewers,

Our revised manuscript highlights changes in blue and includes additional budget/source ablations, controlled selector comparisons, and completed newer-backbone evaluations. We also clarified baseline reproduction, caching policies, latency boundaries, and extraction costs (Secs. 5.1-5.3; App. C). Detailed follow-up responses are available in the reviewer threads.

Thank you for your time. We welcome any remaining questions before the discussion closes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.