acceptodds
Under review as a conference paper at ICLR 2027

RLVU: Recursive Long Video Understanding With Compact Models

Abstract

Using frontier models such as GPT-5 for video question answering can be prohibitively expensive at scale, making cost an important consideration alongside accuracy in applications such as surveillance, online videos, and recorded meetings. This motivates the use of compact reasoning backbones such as GPT-5 mini and Qwen3.5-27B. However, these models often struggle to locate and connect relevant evidence scattered throughout long videos. To address this challenge, we propose RLVU, which combines reusable memory and then recursively reasons over this memory to locate and integrate relevant evidence. Specifically, RLVU constructs video memory through top-down temporal refinement and bottom-up entity consolidation: event level context is propagated to shorter clips, while observations of recurring entities are merged into global records. Given a question, RLVU then performs recursive reasoning over the constructed memory through top-down question decomposition and bottom-up evidence integration. Compared with direct inference using the compact model like GPT-5 mini, the base RLVU configuration improves accuracy by 11.2%, 12.4%, and 4.3% on LongVideoBench, LVBench, and Video-MME, respectively. RLVU also consistently improves accuracy across all three benchmarks when using Qwen3.5-27B as the reasoning backbone, compared with direct inference using the same model. These results demonstrate that structured video memory and recursive reasoning enable compact reasoning models to more effectively locate, verify, and integrate evidence across long videos. The code is at https://anonymous.4open.science/r/RLVU-4B58.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
VLM, Agent, Long Video Understanding

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_968fOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes RLVU, an agentic framework for long-video question answering with compact reasoning backbones. RLVU first builds a question-agnostic memory offline, consisting of hierarchical clip captions and a global entity table. At inference time, a recursive agent retrieves relevant clips from this memory and delegates visual verification to sub-calls, optionally refined by a test-time computing module. The paper reports improvements over direct inference with the same backbones and over agentic baselines on LongVideoBench, LVBench, and Video-MME.

Critical strengths
  1. Consistent gains over direct inference with the same backbone. RLVU improves over direct inference across all three benchmarks and both backbones, with especially large gains on questions that require reasoning across distant parts of a video. This matters because it shows that structured memory combined with recursive retrieval is useful for long-video understanding, even if the source of the gains relative to other agentic methods remains unclear.

  2. A reusable and well-documented system. The memory is question-agnostic and can be reused across queries on the same video. The paper also provides code, prompts, and the execution algorithm, and transparently reports offline construction time. As a result, the system is practical for multi-query settings and easy to reproduce and build upon.

Critical weaknesses
  1. The cost motivation is not supported. The paper motivates compact backbones by the expense of frontier models, yet it never compares against a frontier model in accuracy or end-to-end cost. Its notion of “compact” is subjective: GPT-5 mini is a paid proprietary API, and Qwen3.5-27B requires high-end GPUs. Moreover, the TTC module relies on GPT-5 itself, together with Gemini-3-Flash and Qwen3.6-Plus, directly contradicting the paper’s premise. This is consequential to my assessment because the paper’s central motivation, efficient long-video understanding without frontier models, is neither validated nor consistently followed.

  2. The gains of the pipeline are not disentangled from the backbone. Because the framework mainly combines existing ideas (hierarchical captioning, entity linking, and recursive calls), its value rests on showing that the pipeline itself drives the gains. However, most agentic baselines use older or weaker backbones, and direct inference with Qwen3.5-27B alone already matches or exceeds many of them. In addition, RLVU always builds its memory with Qwen3.5-27B, which strongly affects accuracy, while the memory-construction model of the baselines is not matched. This is the main reason for my rating: without backbone-matched comparisons, it is unclear whether RLVU’s advantage comes from the proposed pipeline or simply from stronger models.

Questions for authors
  1. Can you re-run representative agentic baselines with the same reasoning backbones (GPT-5 mini and Qwen3.5-27B) and the same memory-construction VLM, to separate pipeline gains from backbone gains?

  2. How does RLVU compare with direct inference using a frontier model (e.g., GPT-5), and with memory-based baselines such as DVD, in both accuracy and end-to-end cost (offline memory construction plus online inference)? What criterion makes GPT-5 mini and Qwen3.5-27B “compact”?

8 comments

W1: The cost motivation is not supported. The paper motivates compact backbones by the expense of frontier models, yet it never compares against a frontier model in accuracy or end-to-end cost. Its notion of “compact” is subjective: GPT-5 mini is a paid proprietary API, and Qwen3.5-27B requires high-end GPUs.

Response: Thank you for raising this concern. Our motivation is to make accurate long-video understanding more affordable and accessible by reducing reliance on expensive frontier models. We observe that many recent long-video understanding methods rely on powerful frontier models, such as GPT-5 and Gemini Pro series, to achieve strong reasoning performance. While effective, repeatedly invoking these models can be costly, particularly for popular online videos, educational lectures, surveillance archives, and recorded meetings that may serve many queries. Compact models offer a more economical alternative but often struggle with evidence localization and complex reasoning across long videos. RLVU addresses this gap through coordinated memory construction and recursive reasoning. We clarify our model-selection criteria and provide additional evidence below.

(1) Lower inference cost. We use “compact” in a relative, cost-oriented sense rather than imposing a fixed parameter-count threshold. GPT-5 mini costs \0.25/0.25/\\2.00 per million input/output tokens, compared with \1.25/1.25/\\10.00 for GPT-5, representing an 80% reduction in both token prices [1][2]. Although GPT-5 mini is a proprietary API, its lower pricing makes it a more affordable alternative to frontier models for repeated inference.

(2) More accessible deployment. Qwen3.5-27B is an open-weight model that supports 4-bit quantization [3]. Under suitable context-length and concurrency settings, its quantized weights can fit within 24 GB of GPU memory, making deployment on consumer GPUs such as the RTX 3090/4090 feasible [4]. Our use of H200 GPUs primarily accelerates experiments rather than reflecting a strict hardware requirement. Practical deployment remains subject to KV-cache and runtime memory requirements.

(3) Additional quantization experiment. To further evaluate efficiency, we compare BF16 and 4-bit Qwen3.5-27B for offline memory construction on 20 LVBench videos (21.18 hours, 266 questions), using the same H200 environment and unchanged online reasoning settings.

Table: Effect of 4-bit Quantization on Memory Construction Efficiency and Accuracy

Offline memory modelAccuracy (%)Offline GPU time (hours)Avg. time/video (minutes)
Qwen3.5-27B (BF16)72.5634.6103.8
Qwen3.5-27B (4-bit)71.8026.479.2

Quantization reduces offline GPU time by 23.7%, saving 24.6 GPU-minutes per video, with only a 0.75-percentage-point accuracy decrease (193/266 vs. 191/266). These results show that lower-precision memory construction can improve efficiency while largely preserving downstream accuracy.

(4) Comparison with frontier models. To directly address the cost motivation, we additionally compare RLVU with direct GPT-5 inference and DVD in both accuracy and amortized end-to-end cost (Q2). Despite using GPT-5 mini, base RLVU outperforms direct GPT-5 by 11.4 and 11.6 percentage points on LongVideoBench and LVBench, respectively, while trailing by 2.2 points on Video-MME. Our cost analysis further shows that reusable memory can reduce amortized inference costs in repeated-query settings.

Together, these results support our use of cost-efficient reasoning backbones and clarify that “compact” refers to relative inference cost and deployment accessibility, rather than an absolute model-size threshold.

References

[1] OpenAI. “GPT-5 mini.” OpenAI API Documentation. [https://developers.openai.com/api/docs/models/gpt-5-mini]

[2] OpenAI. “GPT-5.” OpenAI API Documentation. [https://developers.openai.com/api/docs/models/gpt-5]

[3] Qwen Team. “Qwen3.5-27B.” Hugging Face Model Card, 2026. [https://huggingface.co/Qwen/Qwen3.5-27B]

[4] Unsloth. “Qwen3.5: How to Run Locally.” Unsloth Documentation, 2026. [https://unsloth.ai/docs/models/qwen3.5]

W2: The TTC module relies on GPT-5 itself, together with Gemini-3-Flash and Qwen3.6-Plus, directly contradicting the paper’s premise. This is consequential to my assessment because the paper’s central motivation, efficient long-video understanding without frontier models, is neither validated nor consistently followed.

Response: Thank you for raising this concern. We would like to clarify that TTC does not contradict our motivation of reducing reliance on frontier models, for two reasons: (1) Main results are independent of TTC: All primary accuracy and efficiency results are obtained using compact reasoning backbones, without invoking additional frontier models. (2) TTC is disabled by default and serves only as an optional framework extension: It explores how RLVU’s recursive execution can selectively incorporate stronger models for a limited number of difficult, evidence-grounded subproblems, rather than relying on frontier models throughout the entire long-video reasoning process. We explain these points below.

(1) Main results are obtained without TTC. As stated in Secs. 4.1–4.2 and Appendix A.2, all primary accuracy comparisons and online-token analyses use base RLVU with GPT-5 mini or Qwen3.5-27B, without invoking GPT-5, Gemini-3-Flash, or Qwen3.6-Plus. Specifically, without TTC, RLVU achieves 75.9%/71.7%/65.7% with GPT-5 mini and 73.4%/73.1%/68.8% with Qwen3.5-27B on LongVideoBench, LVBench, and Video-MME, respectively. These results demonstrate that RLVU’s main performance gains do not depend on frontier models.

(2) TTC is an optional extension enabled by recursive execution. RLVU’s structured memory and recursive reasoning decompose complex questions into focused subproblems with localized evidence. This provides an opportunity to introduce stronger models only when compact models encounter particularly difficult cases. Rather than using frontier models to process the entire video or handle every reasoning step, TTC focuses their assistance on selected subproblems, while compact models remain responsible for the main reasoning process. This allows RLVU to explore additional reasoning capability without making frontier models a requirement of the framework.

(3) TTC is separate from our primary efficiency claims. TTC explores the potential accuracy improvements available under an additional inference budget. Since stronger-model calls incur extra costs, TTC is evaluated and reported separately from base RLVU. Its results are not included in our primary accuracy or efficiency comparisons, and we do not claim that TTC itself achieves the same cost efficiency as base RLVU.

Overall, RLVU achieves its main results entirely with compact models, while TTC demonstrates the flexibility of its recursive framework to incorporate stronger models selectively when additional resources are available. We will make the default-disabled and optional nature of TTC more explicit in the revised manuscript.

W3: The gains of the pipeline are not disentangled from the backbone. Because the framework mainly combines existing ideas (hierarchical captioning, entity linking, and recursive calls), its value rests on showing that the pipeline itself drives the gains. However, most agentic baselines use older or weaker backbones, and direct inference with Qwen3.5-27B alone already matches or exceeds many of them.

Response: Thank you for raising this concern. RLVU’s improvements are not simply due to stronger backbones, as shown by our new matched-backbone experiments and controlled ablations. We provide three pieces of evidence: (1) new comparisons using the same memory-construction and reasoning models, (2) controlled ablations of memory construction and recursive reasoning, and (3) the methodological differences between RLVU and existing approaches.

(1) New Matched-backbone comparisons. To directly address this concern, we additionally compare RLVU with DVD on the same LVBench subset (20 videos, 266 questions), using Qwen3.5-27B for memory construction and either GPT-5 mini or Qwen3.5-27B for reasoning. Both methods retain their respective memory construction and reasoning procedures. Under these matched settings, RLVU outperforms DVD by 3.7 pp (69.5% vs. 65.8%) with GPT-5 mini and 2.7 pp (72.6% vs. 69.9%) with Qwen3.5-27B. These results show that the improvements cannot be explained solely by stronger models. Detailed results are provided in our response to Q1.

(2) Controlled ablations of memory and reasoning. Tables 2–3 further demonstrate that both memory construction and recursive reasoning contribute to RLVU’s accuracy under controlled settings, rather than relying solely on backbone strength.

  • Memory construction (Table 2): With the backbone held fixed, parent-conditioned temporal refinement improves accuracy by 3.7/1.1 pp, while global entity consolidation improves accuracy by 3.8/2.3 pp on LongVideoBench/LVBench.S. These results support the effectiveness of context-aware memory construction and cross-clip entity association.
  • Recursive reasoning (Table 3): With identical preconstructed memory and GPT-5 mini backbone, recursive reasoning improves accuracy over non-recursive iterative retrieval by 6.8 pp (69.1% → 75.9%) on LongVideoBench and 4.4 pp (65.7% → 70.1%) on LVBench.S. These gains cannot be attributed to differences in memory or backbone selection.

(3) Methodological distinction. RLVU does not simply combine hierarchical captioning, entity linking, and recursive calls. Instead, it coordinates memory construction with recursive reasoning through a global-to-local-to-global design. Top-down temporal refinement creates multi-scale video descriptions that help reasoning subcalls locate relevant evidence, while bottom-up entity consolidation connects observations across distant clips for cross-event reasoning. Recursive reasoning then retrieves, verifies, and integrates this structured evidence through focused subcalls. Unlike methods that design memory and reasoning separately, RLVU aligns how evidence is organized with how it is accessed during reasoning, reducing both evidence-search and reasoning complexity for compact models.

Together, the matched-backbone comparisons and controlled ablations demonstrate that RLVU’s gains come from its framework design rather than backbone strength alone.

Reviewer_xsj5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RLVU for long-video QA with compact reasoning models. It builds reusable question-independent video memory through multi-scale temporal refinement and entity consolidation, then performs recursive retrieval, local visual verification, and evidence integration. Experiments on LongVideoBench, LVBench, and Video-MME show consistent improvements with GPT-5 mini and Qwen3.5-27B.

Critical strengths
  1. RLVU consistently improves over direct compact-model inference across three long-video benchmarks and two reasoning backbones.
  2. The question-independent memory can be reused across multiple queries, and the ablations support the contributions of temporal hierarchy and entity consolidation.
Critical weaknesses
  1. Limited novelty of the recursive reasoning component. The method combines several existing ideas, and the recursive execution closely follows prior recursive-agent frameworks. Moreover, (D_{\max}=1) performs best, making the role of deeper recursion unclear.
  2. Unmatched visual/computational budget. Direct baselines use 256 sampled frames, while RLVU performs dense multi-scale offline processing and entity construction. Therefore, part of the gain may come from substantially more visual processing rather than the proposed reasoning mechanism itself.
Questions for authors
  1. Can the authors provide a matched visual/computational budget comparison?

  2. What is the main methodological novelty of RLVU’s recursive reasoning beyond existing recursive-agent execution?

  3. What is the break-even number of queries per video when offline memory construction cost is included?

7 comments

W1: Limited novelty of the recursive reasoning component. The method combines several existing ideas, and the recursive execution closely follows prior recursive-agent frameworks. Moreover, (D_{\max}=1) performs best, making the role of deeper recursion unclear.

Response: Thank you for raising this concern. We agree that recursive execution itself is not new. However, existing recursive-agent frameworks primarily reduce task complexity through decomposition, while evidence retrieval and cross-event reasoning remain challenging for compact models handling long videos. RLVU addresses these limitations by coordinating recursive reasoning with structured video memory, reducing both evidence-search and reasoning complexity. We clarify the methodological distinction and the role of recursion depth below.

(1) Novelty of recursive reasoning. Rather than simply applying general recursive execution to video inputs, RLVU co-designs recursive memory construction and reasoning to make each subcall more focused and manageable. Specifically, (a) Memory-guided evidence localization: Top-down temporal refinement provides multi-scale, context-aware descriptions that help recursive subcalls locate relevant events without searching the entire video. (b) Entity-aware evidence integration: Bottom-up entity consolidation links observations across distant clips, allowing recursive reasoning to retrieve and connect evidence involving recurring entities. (c) Selective visual verification: When memory descriptions are insufficient, subcalls inspect the corresponding original frames and return concise observations for bottom-up integration. Together, these mechanisms align evidence organization with recursive execution, allowing compact models to resolve focused subproblems rather than simultaneously handling global evidence search and complex reasoning.

(2) Effectiveness of shallow recursion. Importantly, Dmax⁡=1D_{\max}=1 limits nesting depth, not the number of subcalls. A root call can invoke multiple focused subcalls to retrieve and verify evidence from different video segments before integrating their observations. With identical memory and GPT-5 mini backbone, enabling recursion (Dmax⁡=1D_{\max}=1) improves accuracy over non-recursive iterative reasoning (Dmax⁡=0D_{\max}=0) by 6.8 pp on LongVideoBench (69.1% → 75.9%) and 4.4 pp on LVBench.S (65.7% → 70.1%) (Table 3). These results show that even shallow recursion provides substantial benefits beyond iterative retrieval.

(3) Role of deeper recursion. Deeper recursion is intended for subproblems that cannot be resolved through a single level of evidence retrieval and verification and require further decomposition. However, RLVU’s structured memory already helps localize relevant events and connect related observations, allowing many subproblems to be resolved within one recursive level. Additional nesting therefore offers limited opportunities for further simplification while introducing more intermediate reasoning steps. Consistent with this trade-off, Dmax⁡=1D_{\max}=1 achieves the highest accuracy in our experiments, while deeper recursion provides no further gains. Similar depth-dependent behavior is also reported in the updated RLM study [1].

Overall, RLVU’s contribution lies in aligning recursive reasoning with structured video memory, rather than introducing a new recursion primitive or pursuing deeper execution trees.

Reference

[1] Zhang, Alex L., Tim Kraska, and Omar Khattab. “Recursive Language Models.” arXiv:2512.24601, 2025.

W2: Unmatched visual/computational budget. Direct baselines use 256 sampled frames, while RLVU performs dense multi-scale offline processing and entity construction. Therefore, part of the gain may come from substantially more visual processing rather than the proposed reasoning mechanism itself.

&

Q1: Can the authors provide a matched visual/computational budget comparison?

Response: Thank you for raising this concern. We agree that RLVU uses additional offline visual processing. However, our results show that its accuracy gains cannot be attributed solely to a larger visual budget: increasing direct inference to 1,500 frames still leaves a substantial accuracy gap, while recursive reasoning provides additional improvements under identical offline memory. We provide two complementary comparisons below.

(1) Higher-budget direct inference. Our original 256-frame baseline follows established settings in LongVideoBench [1], DVD [2], and LensWalk [3]. To address this concern, we extend direct GPT-5 mini inference on LVBench to 512, 1,024, and 1,500 frames, adjusting resolution to fit the context window. Increasing direct inference from 256 to 1,500 frames improves accuracy by 6.1 pp, yet RLVU still leads by 6.3 pp (71.7% vs. 65.4%) with approximately 54% lower online inference cost. Higher frame–resolution configurations exceeded the tested context limit, showing the practical constraints of scaling single-pass visual input.

Table I: Accuracy and Online Cost under Different Visual Input Budgets (LVBench).

MethodFramesResolutionAvg. input tokensOnline cost/queryAccuracy (%)
Direct GPT-5 mini2561280×720281,130$0.07159.3
Direct GPT-5 mini512896×504275,537$0.07060.3
Direct GPT-5 mini1,024640×360296,016$0.07461.6
Direct GPT-5 mini1,500544×306308,511$0.07665.4
RLVU (GPT-5 mini)———$0.03571.7

(2) Reasoning gains with identical offline memory. More importantly, Table 3 isolates the reasoning contribution by keeping the preconstructed memory and GPT-5 mini backbone unchanged. Compared with non-recursive iterative reasoning (Dmax⁡=0D_{\max}=0), recursive reasoning (Dmax⁡=1D_{\max}=1) improves accuracy by 6.8 pp on LongVideoBench (69.1% → 75.9%) and 4.4 pp on LVBench.S (65.7% → 70.1%). Since both settings use identical offline visual information, these gains cannot be explained by additional memory construction or denser preprocessing.

Together, these comparisons demonstrate that RLVU’s improvements stem not only from richer visual evidence but also from how structured memory and recursive reasoning organize, verify, and integrate that evidence for compact models.

References:

[1] Wu, Haoning, et al. "Longvideobench: A benchmark for long-context interleaved video-language understanding." Advances in Neural Information Processing Systems 37 (2024): 28828-28857.

[2] Zhang, Xiaoyi, et al. "Deep video discovery: Agentic search with tool use for long-form video understanding." Advances in Neural Information Processing Systems 38 (2026): 89863-89895.

[3] Li, Keliang, et al. "LensWalk: Agentic video understanding by planning how you see in videos." arXiv preprint arXiv:2603.24558 (2026).

[4] Pang, Ziqi, and Yu-Xiong Wang. "Mr. video:" mapreduce" is the principle for long video understanding." arXiv preprint arXiv:2504.16082 (2025).

[5] Fu, Chaoyou, et al. "Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025.

Q2: What is the main methodological novelty of RLVU’s recursive reasoning beyond existing recursive-agent execution?

Response: Thank you for this question. Existing recursive-agent frameworks, such as RLM, reduce task complexity through recursive decomposition. However, applying them to long-video understanding with compact models remains challenging: (1) decomposing a question does not necessarily reduce the effort needed to locate relevant evidence across long videos, and (2) resolving individual subquestions does not automatically establish relationships between events occurring far apart. To address these challenges, RLVU co-designs recursive memory construction and reasoning to reduce both evidence-search and reasoning complexity. The key distinctions are as follows.

(1) Memory-guided recursive reasoning. Unlike general recursive agents that primarily focus on task decomposition, RLVU coordinates recursive reasoning with structured video memory. Top-down temporal refinement produces context-aware descriptions at multiple scales, allowing reasoning subcalls to narrow their search to relevant events. Meanwhile, bottom-up entity consolidation connects observations across distant clips, helping subcalls retrieve related evidence for cross-event reasoning. In this way, memory construction directly supports the evidence retrieval and integration required by recursive reasoning, rather than serving as a separate preprocessing step.

(2) Evidence-grounded recursive execution. Rather than simply decomposing questions and combining subtask answers, RLVU integrates evidence retrieval and visual verification into recursive execution. Each subcall retrieves relevant memory, selectively inspects original frames when necessary, and returns concise observations for bottom-up integration. This allows compact models to focus on localized evidence and manageable subproblems instead of processing extensive video content and complex relationships simultaneously.

(3) Experimental evidence and recursion depth. With identical memory and GPT-5 mini backbone (Table 3), recursive reasoning (Dmax⁡=1D_{\max}=1) improves accuracy over non-recursive iterative retrieval (Dmax⁡=0D_{\max}=0) by 6.8 pp on LongVideoBench (69.1% → 75.9%) and 4.4 pp on LVBench.S (65.7% → 70.1%). Importantly, Dmax⁡=1D_{\max}=1 still allows multiple subcalls to retrieve and verify evidence from different video segments. Since structured memory already supports evidence localization, further nesting provides limited additional benefits in our evaluated setting.

Overall, RLVU’s contribution lies not in introducing a new recursion mechanism, but in aligning recursive reasoning with structured video memory, enabling compact models to locate, verify, and integrate evidence more effectively.

Reviewer_gHXXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper is about doing long video QA without relying on a big frontier model for every question. The basic problem is familiar: compact models like GPT-5 mini and Qwen are cheaper but they miss things in long videos, especially when the evidence is spread out or the same person/object keeps reappearing in different parts. The proposed method RLVU builds a memory for the video first. It does this at three time scales with the broader event description passed down to the shorter clips, and it also merges observations of the same entity across clips into one global record. At question time, it then works recursively, breaking the question into smaller sub-questions, pulling the relevant clips from that memory checking the actual frames locally and putting the answers back together. The results are fairly convincing. They test on LongVideoBench, LVBench and Video-MME and get consistent gains over just running the same model directly, around 11 points on LongVideoBench and LVBench for GPT-5 mini and a smaller gain on Video-MME. From the ablations study: removing the finer temporal levels, the entity consolidation and the recursion hurts performance.

Critical strengths
  • The gains are consistent and not tied to one model setups. RLVU improves over direct use of the same backbone on all three benchmarks and with both backbones. It also beats the reproduced DVD baseline where the backbone was matched. That suggests the memory and recursion are doing the work, rather than being a result that only holds on one dataset or with one model.
  • The ablations actually back up the e2e design. Adding the finer temporal levels helps a lot, consolidated entity memory does better than just keeping local entity records, and adding one level of recursion improves over iterative retrieval alone. The large gain on Relation questions also fits what the method is supposed to be good at, connecting evidence across time.
Critical weaknesses

The cost claim is not proven and cost is the main motivation for the paper. The reported savings are online tokens compared to DVD, and they exclude memory construction. That construction is not cheap, about 52 minutes on an H200 for a 30 minute video and about 98 minutes for a 60 minute video in their own breakdown. In that case the memory pays for itself depends on how many questions are asked per video and that total cost per question is never reported for any benchmark.

Questions for authors

Clarification questions:

  • What is the total cost per question once memory construction and indexing are included, amortised at the actual number of questions per video in each benchmark? Please give this for RLVU, direct GPT-5 mini, and DVD.
  • How does direct inference perform if it is given a similar token budget to RLVU, for example through more frames or repeated sampling?

Request: Could you report confidence intervals for the main ablations?

6 comments
W1: The cost claim is not proven and cost is the main motivation for the paper. The reported savings are online tokens compared to DVD, and they exclude memory construction. That construction is not cheap, about 52 minutes on an H200 for a 30 minute video and about 98 minutes for a 60 minute video in their own breakdown.

Response: Thank you for raising this concern. RLVU improves long-video reasoning accuracy with compact models while amortizing memory construction costs across repeated queries. To quantify this trade-off, we additionally measure DVD’s preprocessing overhead and compare accuracy and total amortized cost, including offline construction and online inference.

(1) Total Cost Model. For a video queried QQ times, the amortized cost per question is: CRLVU(Q)=CbuildQ+Cˉonline.C_{\mathrm{RLVU}}(Q)=\frac{C_{\mathrm{build}}}{Q}+\bar{C}_{\mathrm{online}}. On the same sampled videos and H200 GPU, DVD requires 41.8/93.3 minutes of preprocessing for 30-/60-minute videos, versus 52.46/98.29 minutes for RLVU. At $3.59/GPU-hour, the corresponding costs are \2.50/2.50/\\5.58 and \3.14/3.14/\\5.88, respectively.

(2) Accuracy–Cost Comparison. Existing benchmarks primarily evaluate reasoning accuracy rather than repeated-query workloads. LongVideoBench Long contains 564 questions over 188 videos (3.00/video), LVBench 1,549 over 103 (15.04/video), and Video-MME Long 900 over 300 (3.00/video).

Real-world query volumes (Q values). In practice, the same video may receive hundreds of queries: online lectures serve questions from many students, recorded meetings are revisited for decisions and assignments, and surveillance archives support repeated investigations. Since memory is constructed once, its cost can be shared across queries. We therefore evaluate both the observed benchmark query densities (Q=ActualQ=\mathrm{Actual}) and illustrative reuse levels (Q=100/200Q=100/200).

Table I: Accuracy and Amortized Cost under Different Query Volumes.

Benchmark (Videos / Questions)MethodAccuracy (%)Q=Actual (30/60 min)Q=100 (30/60 min)Q=200 (30/60 min)
LongVideoBench (188 / 564)Direct GPT-5 mini64.7$0.076$0.076$0.076
DVD67.7\0.877/0.877 / \\1.904\0.068/0.068 / \\0.099\0.056/0.056 / \\0.071
RLVU75.9\1.084/1.084 / \\1.998$0.069 / \$0.097\0.054/0.054 / \\0.067
LVBench (103 / 1,549)Direct GPT-5 mini59.3$0.071$0.071$0.071
DVD69.6\0.227/0.227 / \\0.432\0.086/0.086 / \\0.117\0.074/0.074 / \\0.089
RLVU71.7\0.242/∗∗0.242 / **\\0.424**\0.064/0.064 / \\0.092\0.049/0.049 / \\0.062
Video-MME (300 / 900)Direct GPT-5 mini61.4$0.070$0.070$0.070
DVD58.3\0.892/0.892 / \\1.919\0.083/0.083 / \\0.114\0.071/0.071 / \\0.086
RLVU65.7\1.086/1.086 / \\2.000\0.071/0.071 / \\0.099\0.056/0.056 / \\0.069

Note: Costs are in USD/query. Paired values use 30-/60-minute reference preprocessing costs. Q=Actual reflects each benchmark’s average questions per video; Q=100/200 are illustrative reuse levels. DVD/RLVU costs include preprocessing and online inference. Direct GPT-5 mini uses 256 frames.

Three key findings emerge: (a) Higher accuracy and lower online costs. RLVU outperforms DVD by 8.2/2.1/7.4 percentage points while reducing online costs by 12%/46%/31% across the three benchmarks. (b) Lower amortized costs through reuse. At LVBench’s observed query density, RLVU already costs less than DVD for 60-minute videos (\0.424vs.0.424 vs. \\0.432). At Q=200Q=200, RLVU achieves both higher accuracy and lower amortized costs than DVD across all benchmarks. (c) Cost advantage over direct inference. Against uncached GPT-5 mini using 1,024 frames (\0.0745/query),RLVUbreaksevenafterapproximately80/149queriesfor30−/60−minutevideos,reaching0.0745/query), RLVU breaks even after approximately 80/149 queries for 30-/60-minute videos, reaching \\0.051/\0.064perqueryat0.064 per query at Q=200$.

Overall, RLVU combines higher accuracy with lower online inference costs, making its upfront memory construction increasingly economical as query volume grows.

W2: In that case the memory pays for itself depends on how many questions are asked per video and that total cost per question is never reported for any benchmark.

&

Q1:What is the total cost per question once memory construction and indexing are included, amortised at the actual number of questions per video in each benchmark? Please give this for RLVU, direct GPT-5 mini, and DVD.

Response: Thank you for this question. We agree that evaluating RLVU’s cost efficiency requires accounting for both offline memory construction and online inference, rather than online token consumption alone. Accordingly, we extend our analysis to report the total amortized cost per question for RLVU, DVD, and direct GPT-5 mini under different query volumes, as detailed in Table I of our response to W1. The results show that RLVU achieves higher accuracy with lower online inference costs than DVD, while its upfront preprocessing cost can be progressively recovered through memory reuse.

(1) Total cost at actual benchmark query densities. Table I reports the amortized cost per question for all three methods, using the actual 3.00/15.04/3.00 questions per video on LongVideoBench, LVBench, and Video-MME, respectively. The estimates combine measured preprocessing costs with benchmark-specific online inference costs, providing the requested total-cost comparison.

(2) Accuracy–cost trade-off with compact models. Simply replacing frontier models with compact backbones can substantially reduce long-video QA accuracy. For example, VideoSeek achieves 73.5% with GPT-5 but only 55.5% with GPT-5 mini (−18.0 pp) on LongVideoBench. Similarly, direct GPT-5 mini achieves 64.7%/59.3%/61.4% across the three benchmarks, while DVD with GPT-5 mini achieves 67.7%/69.6%/58.3%. In contrast, RLVU achieves 75.9%/71.7%/65.7% with GPT-5 mini, while reducing online inference costs by 12%/46%/31% compared with DVD. These results highlight the importance of structured memory and recursive reasoning in improving the accuracy–cost trade-off of compact models.

(3) Benefits of memory reuse. At LVBench’s actual query density, RLVU already achieves a lower amortized cost than DVD for 60-minute videos (\0.424vs.0.424 vs. \\0.432). At Q=200Q=200, RLVU achieves both higher accuracy and lower amortized costs than DVD across all three benchmarks. Compared with direct GPT-5 mini using 1,024 frames, RLVU reaches monetary break-even after approximately 80/149 queries for 30-/60-minute videos.

Q2: How does direct inference perform if it is given a similar token budget to RLVU, for example through more frames or repeated sampling?

Response: Thank you for this suggestion. We agree that increasing the visual input budget through more frames or repeated sampling is an important comparison. We therefore extend direct GPT-5 mini inference from 256 to 1,500 uniformly sampled frames and further analyze repeated sampling. Our results show that denser sampling improves accuracy from 59.3% to 65.4% on LVBench, but RLVU still achieves 71.7%, outperforming direct inference by 6.3 percentage points with approximately 54% lower online inference cost.

(1) Experimental Settings. Our original 256-frame baseline follows established long-video evaluation settings, including LongVideoBench [1] and LensWalk [2], while VideoSeek evaluates direct GPT-5 using 384 uniformly sampled frames [3]. To provide a stronger comparison, we evaluate additional frame-count and resolution configurations, adjusting resolution to accommodate more frames within the available context budget. We also report configurations rejected due to context limits.

Table II: Accuracy and Online Cost under Different Visual Input Configurations (LVBench).

FramesResolutionAvg. input tokensOnline cost/queryAccuracy (%)
2561280×720281,130$0.07159.3
2961280×720Context limit——
512896×504275,537$0.07060.3
512976×549Context limit——
1,024640×360296,016$0.07461.6
1,024688×387Context limit——
1,500544×306308,511$0.07665.4
RLVU (GPT-5 mini)——$0.03571.7

(2) Results and Analysis.

(a) More frames. Increasing uniformly sampled frames from 256 to 1,500 improves direct-inference accuracy by 6.1 percentage points, but RLVU still leads by 6.3 points (71.7% vs. 65.4%). Moreover, RLVU reduces online inference cost by approximately 54% (\0.035vs.0.035 vs. \\0.076/query). Higher-resolution configurations exceed the tested context limit, indicating that further increasing visual coverage within a single model call is constrained by the available context budget.

(b) Repeated sampling. Repeated sampling primarily increases visual coverage but does not inherently improve how compact models locate and connect relevant evidence across distant video segments. Our experiments show that increasing uniformly sampled frames by nearly 6× (256 to 1,500) improves accuracy by only 6.1 points, still trailing RLVU by 6.3 points. Moreover, multiple independent sampling passes would introduce additional inference costs and require cross-pass evidence integration, which is itself a key challenge in long-video reasoning. In contrast, RLVU explicitly addresses this challenge through structured memory and query-driven recursive reasoning, enabling targeted evidence retrieval and integration rather than repeated uniform sampling.

These results indicate that increasing visual coverage alone is insufficient to match RLVU’s accuracy. RLVU instead organizes and connects evidence across long videos, enabling compact models to achieve stronger reasoning performance without repeatedly processing the entire video.

References

[1] Wu, Haoning, et al. “LongVideoBench: A Benchmark for Long-Context Interleaved Video-Language Understanding.” NeurIPS, 2024.

[2] Li, Keliang, et al. “LensWalk: Agentic Video Understanding by Planning How You See in Videos.” arXiv:2603.24558, 2026.

[3] Lin, Jingyang, et al. “VideoSeek: Long-Horizon Video Agent with Tool-Guided Seeking.” arXiv:2603.20185, 2026.

Reviewer_AEeQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RLVU which constructs video memory through top-down temporal refinement and bottom-up entity consolidation. It performs recursive reasoning over the constructed memory through top-down question decomposition and bottom-up evidence integration. The evaluation results show RLVU has achieved remarkable accuracy improvements on VQA benchmarks.

Critical strengths

1.This paper clearly presents RLVU with detailed figures. 2.The introduction of the method is concise and clear.

Critical weaknesses

1.The discussion on time cost is unclear. For example, existing end-to-end VQA methods only use hundreds of frames to answer the question, what about RLVU? It seems a lot of frames are involved to compute for accuracy improvements. The efficiency of RLVU is the most important concern. 2.The entire method is more like a pipeline to manage and use memory for VQA. But during memory management, this paper lacks an effective algorithm to control and optimize the entire process. 3.The comparison of VQA methods are kind of outdated. There are lots of VQA methods emerging in 2026, such as VideoSeek (https://arxiv.org/pdf/2603.20185), EGAgent (https://arxiv.org/pdf/2601.18157). So the so-called improvement in accuracy is not all that convincing.

Questions for authors

see the weakness

5 comments
W1: The discussion on time cost is unclear. For example, existing end-to-end VQA methods only use hundreds of frames to answer the question, what about RLVU? It seems a lot of frames are involved to compute for accuracy improvements. The efficiency of RLVU is the most important concern.

Response: Thank you for raising this important concern. Our key insight is to enable accurate and cost-effective long-video understanding by combining compact models with reusable structured memory and recursive reasoning. Our cost advantages come from two aspects: (1) Offline: Video memory is constructed once and reused across multiple queries, amortizing preprocessing costs. (2) Online: Compact models selectively retrieve and verify relevant evidence, keeping per-query inference costs low. Our analysis shows that RLVU reaches monetary break-even after approximately 80 and 149 queries for 30- and 60-minute videos, respectively, compared with E2E GPT-5 mini using 1,024 frames per query. We provide details below.

(1) Representative Application Scenarios. RLVU is particularly valuable in three scenarios involving repeated queries over the same video, making memory reuse particularly valuablediscussed in our Introduction: (i) Surveillance archives, where long recordings may be queried repeatedly to track individuals, locate events, and reconstruct activities across timestamps; (ii) Online videos, where popular lectures, tutorials, and documentaries may receive hundreds or thousands of questions from different users; and (iii) Recorded meetings, where team members revisit recordings to retrieve decisions, track discussions, and identify task assignments.

(2) Detailed Efficiency Analysis. We evaluate RLVU’s efficiency by accounting for both one-time memory construction and per-query inference. For a video queried QQ times, the amortized cost per query is: CRLVU(Q)=CofflineQ+Cˉonline.C_{\mathrm{RLVU}}(Q)=\frac{C_{\mathrm{offline}}}{Q}+\bar{C}_{\mathrm{online}}.

  • Visual processing and offline cost. As described in Appendix A.2, processing a 30-minute video involves approximately 3,600 sampled frames and 8,550 cumulative frame inputs across memory construction stages. According to Table 8, preprocessing takes 52.46 minutes for a 30-minute video and 98.29 minutes for a 60-minute video on a single NVIDIA H200 GPU. At a reference H200 rental rate of \3.59/hour[1],thecorrespondingofflinecostsareapproximately3.59/hour [1], the corresponding offline costs are approximately \\3.14 and \$5.88, respectively. These costs are incurred only once per video.

  • Accuracy and online inference cost. To examine whether RLVU’s accuracy gains simply come from processing more frames, we further evaluate E2E GPT-5 mini with different frame budgets on LVBench. As shown below, increasing the input from 256 to 1,500(the maximum supported by the API) frames improves E2E accuracy from 59.3% to 65.4%, but still falls 6.3 percentage points below RLVU (71.7%). These results show that increasing the visual input budget alone is insufficient to match RLVU’s accuracy. Instead, structured memory and recursive reasoning enable compact models to locate and integrate relevant evidence more effectively. Moreover, once memory is constructed, RLVU requires only 8.86 additional frame inputs per query on average, with an online inference cost of $0.035 per query, approximately 54% lower than E2E inference with 1,500 frames.

Table: Accuracy and Online Inference Cost under Different Frame Budgets.

MethodFramesResolutionAvg. input tokensOnline cost/queryAccuracy
E2E GPT-5 mini2561280×720281,130$0.07159.3%
E2E GPT-5 mini512896×504275,537$0.07060.3%
E2E GPT-5 mini1,024640×360296,016$0.07461.6%
E2E GPT-5 mini1,500544×306308,511$0.07665.4%
RLVU (GPT-5 mini)———$0.03571.7%
  • Amortized cost and break-even analysis. Using the above cost model, we calculate the average cost per query as video memory is reused across multiple questions. At 200 queries, RLVU’s amortized cost decreases to \0.051and0.051 and \\0.064 per query for 30- and 60-minute videos, respectively, compared with 0.07450.0745 for E2E inference. These results show that RLVU can achieve higher accuracy and lower amortized costs for frequently queried videos, rather than merely trading additional computation for accuracy.

Table: Amortized Inference Cost and Break-even Analysis.

Video durationOffline costRLVU online/queryE2E/queryRLVU at Q=200Break-even
~30 min$3.14$0.035$0.0745$0.051~80
~60 min$5.88$0.035$0.0745$0.064~149

Note: The E2E baseline uses GPT-5 mini with 1,024 frames per query, without cross-query caching. Costs are in USD.

  • Potential for further cost reduction. Our current preprocessing uses an H200 GPU without optimizing for cost. Since preprocessing is offline, cheaper GPUs may further reduce expenses. For example, assuming an L40S costs \1.20/hourandtakestwiceaslong,thepreprocessingcostfora30−minutevideowoulddropfrom1.20/hour and takes twice as long, the preprocessing cost for a 30-minute video would drop from \\3.14 to \$2.10, reducing the break-even point from 80 to 53 queries. This illustrative example suggests further room for cost optimization of using compact models.

(3) Summary. RLVU trades higher one-time preprocessing costs for reusable video understanding and inexpensive online reasoning. Our results show that this design can achieve a favorable accuracy–cost trade-off for frequently queried videos, while leaving further opportunities to reduce offline costs through more economical hardware.

References

[1] https://www.runpod.io/gpu-models/h200

[2] https://openrouter.ai/openai/gpt-5-mini


W2: The entire method is more like a pipeline to manage and use memory for VQA. But during memory management, this paper lacks an effective algorithm to control and optimize the entire process.

Response: Thank you for this comment. RLVU addresses two key limitations of compact models in long-video understanding: locating scattered evidence and reasoning over complex relationships. To this end, RLVU introduces two complementary recursive designs: (1) structured memory construction, which organizes video evidence across temporal scales and recurring entities, and (2) query-driven recursive reasoning, which decomposes complex questions and integrates verified evidence. Together, they reduce the amount of evidence and reasoning complexity handled by each model call.

(1) Recursive Memory Construction (Sec. 3.1). The two recursive directions follow distinct information dependencies. First, top-down temporal refinement propagates broader event context to finer clips, preventing local descriptions from losing their surrounding context. Then, bottom-up entity consolidation recursively merges observations of recurring entities across clips based on semantic compatibility, type consistency, and temporal continuity. This ordering allows entity consolidation to operate on context-aware local observations rather than isolated descriptions, preserving both event continuity and entity consistency. We have added detailed pseudocode for memory construction in Appendix A.2 to clarify its recursive execution and control flow.

(2) Query-driven Recursive Reasoning (Sec. 3.2). Unlike fixed retrieval pipelines, RLVU dynamically determines which evidence to retrieve, when to verify visual details, and whether further decomposition is necessary. Each recursive call operates on selected memory and a focused subquestion, returning only its observations to the parent. This avoids exposing compact models to the entire video memory and reduces the complexity of individual reasoning steps. The execution algorithm is provided in Appendix A.1.

(3) Experimental Validation. Our ablations validate both designs. On LongVideoBench, parent conditioning, hierarchical retrieval, and entity consolidation improve accuracy by 3.7, 2.4, and 3.8 percentage points, respectively (Table 2). Furthermore, recursive reasoning improves accuracy from 69.1% to 75.9% (+6.8 pp) compared with iterative retrieval without recursion (Table 3). These results demonstrate that RLVU’s improvements stem from structured memory organization and query-driven recursive control, rather than simply adding processing stages.

W3: The comparison of VQA methods are kind of outdated. There are lots of VQA methods emerging in 2026, such as VideoSeek (https://arxiv.org/pdf/2603.20185), EGAgent (https://arxiv.org/pdf/2601.18157). So the so-called improvement in accuracy is not all that convincing.

Response: Thank you for highlighting these recent studies. We have expanded our evaluation to include VideoSeek, EGAgent, and LensWalk[1], covering recent agentic video understanding methods from 2026. Importantly, RLVU consistently outperforms VideoSeek across all three benchmarks under matched reasoning backbones, while also achieving stronger results than LensWalk on LongVideoBench and LVBench despite using more affordable compact models.

(1) VideoSeek: Matched-backbone comparison. We reproduce VideoSeek using the same reasoning backbones as RLVU (GPT-5 mini and Qwen3.5-27B). With GPT-5 mini, RLVU achieves 75.9% vs. 55.5% on LongVideoBench, 71.7% vs. 51.1% on LVBench, and 65.7% vs. 62.6% on Video-MME. Similar improvements are observed with Qwen3.5-27B, confirming that RLVU’s gains are not simply due to differences in reasoning model capacity.

(2) EGAgent: Published results. We include EGAgent’s reported 74.1% on Video-MME Long. Since its precomputed video representations are not publicly available and reconstructing them requires substantial computation, we report the published result rather than a matched-backbone reproduction. Notably, EGAgent uses Gemini 2.5 Pro and audio transcripts, whereas RLVU evaluates Video-MME without subtitles or transcripts. We therefore present this result for reference rather than as a directly comparable setting.

(3) LensWalk: Frontier-model comparison. We additionally include LensWalk, a recent active video observation framework. Despite using compact reasoning models, RLVU achieves 75.9% on LongVideoBench and 71.7% on LVBench, outperforming LensWalk with GPT-5 (68.8%, 66.9%) and o3 (70.6%, 68.6%). These results further support the effectiveness of structured memory and recursive reasoning beyond simply using stronger models.

Table: Comparison with Recent Video Understanding Methods.

MethodMemory modelReasoning modelLongVideoBenchLVBenchVideo-MME
VideoSeek†—Qwen3.5-27B57.658.262.1
VideoSeek†—GPT-5 mini55.551.162.6
VideoSeek—GPT-573.568.470.1
EGAgentLLaVA-Video-7B + GPT-4.1Gemini 2.5 Pro——74.1
LensWalk—GPT-568.866.969.2
LensWalk—o370.668.671.4
RLVU (ours)Qwen3.5-27BQwen3.5-27B73.473.168.8
RLVU (ours)Qwen3.5-27BGPT-5 mini75.971.765.7

† Results reproduced by us. Other baseline results follow their published configurations.

[1] Li, Keliang, et al. "LensWalk: Agentic video understanding by planning how you see in videos." arXiv preprint arXiv:2603.24558 (2026).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Methodology, Comparisons, and Costs

We sincerely thank all reviewers for their careful reading and constructive feedback. Below, we address several concerns shared across multiple reviews and summarize our key findings from additional experiments and analyses. Detailed responses to individual comments are provided in the reviewer-specific sections that follow.

Q1: Methodological Novelty: Global-to-Local-to-Global Co-design for Compact Models (Reviewers AEeQ, xsj5, and 968f).

Compact models offer a more affordable alternative to frontier models for long-video understanding, but often struggle to locate scattered evidence, connect distant events, and perform complex reasoning within a single context. Existing approaches often design video memory construction and reasoning separately, leaving these challenges unresolved even when recursive task decomposition is applied. RLVU addresses these limitations through a global-to-local-to-global design that aligns recursive memory construction with recursive reasoning, reducing both evidence-search and reasoning complexity for compact models. Specifically, (i) Global-to-local: Memory construction progressively refines global video context into multi-scale temporal descriptions, while reasoning decomposes complex questions into focused subproblems and uses this hierarchy to locate relevant evidence. (ii) Local-to-global: Memory construction consolidates local observations into global entity records, while reasoning integrates evidence from focused subcalls to resolve cross-event relationships. This alignment allows compact models to operate on focused evidence and manageable subproblems rather than handling global video search and complex reasoning simultaneously. With identical memory and GPT-5 mini backbone, recursive reasoning improves accuracy by 6.8 and 4.4 percentage points on LongVideoBench and LVBench.S, respectively.

Q2: Fairness of Baseline Comparisons: New Baselines and Higher Visual Budgets (Reviewers AEeQ, gHXX, xsj5, and 968f).

To address concerns about comparison fairness, we have added three recent baselines (VideoSeek, EGAgent, and LensWalk) and extended direct inference from 256 to 1,500 frames, while reproducing relevant methods with matched reasoning backbones where feasible. Increasing direct GPT-5 mini inference to 1,500 frames improves LVBench accuracy from 59.3% to 65.4%, yet RLVU still achieves 71.7%, maintaining a 6.3-percentage-point advantage. Moreover, matched-backbone comparisons with VideoSeek confirm that RLVU’s improvements are not simply due to using stronger reasoning models. These results demonstrate that structured memory and recursive reasoning enable compact models to achieve stronger long-video understanding beyond increased visual coverage, although the comparisons do not fully match total offline and online computation.

Q3: Offline and Online Cost Concerns: Amortized Costs and Break-even Analysis (Reviewers AEeQ, gHXX, xsj5, and 968f).

We acknowledge that direct inference can be more economical for short or infrequently queried videos, where one-time processing avoids the overhead of memory construction. However, RLVU targets long videos with repeated queries, where it achieves higher accuracy and lower amortized costs through reusable memory. This setting covers a broad range of applications, including popular online videos, educational lectures, documentaries, surveillance archives, recorded meetings, and sports replays. Popular videos can attract thousands or even millions of viewers, creating opportunities for query volumes well beyond the break-even threshold. At a reference H200 rental rate of \3.59/hour,memoryconstructioncostsapproximately3.59/hour, memory construction costs approximately \\3.14/\5.88for30−/60−minutevideos,whileonlineinferencecosts5.88 for 30-/60-minute videos, while online inference costs \\0.035 per query. Compared with uncached direct GPT-5 mini using 1,024 frames per query, RLVU reaches monetary break-even after approximately 80/149 queries, respectively. For example, a video with 10,000 viewers would generate 200 queries if just 2% asked one question each, exceeding both thresholds. Thus, direct inference remains attractive for short or rarely queried videos, while RLVU offers stronger accuracy and increasingly favorable amortized costs for frequently accessed long videos. These estimates exclude separately billed indexing charges.

We have revised the manuscript accordingly, including the amortized cost analysis and updated token accounting (Sec. 4.4), additional memory construction details (Appendix A.2), and expanded baseline evaluations and cost comparisons (Appendix B.2–B.5). All revisions are highlighted in orange for clarity. Detailed responses to individual comments are provided below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.