- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes RLVU, an agentic framework for long-video question answering with compact reasoning backbones. RLVU first builds a question-agnostic memory offline, consisting of hierarchical clip captions and a global entity table. At inference time, a recursive agent retrieves relevant clips from this memory and delegates visual verification to sub-calls, optionally refined by a test-time computing module. The paper reports improvements over direct inference with the same backbones and over agentic baselines on LongVideoBench, LVBench, and Video-MME.
Critical strengths
-
Consistent gains over direct inference with the same backbone. RLVU improves over direct inference across all three benchmarks and both backbones, with especially large gains on questions that require reasoning across distant parts of a video. This matters because it shows that structured memory combined with recursive retrieval is useful for long-video understanding, even if the source of the gains relative to other agentic methods remains unclear.
-
A reusable and well-documented system. The memory is question-agnostic and can be reused across queries on the same video. The paper also provides code, prompts, and the execution algorithm, and transparently reports offline construction time. As a result, the system is practical for multi-query settings and easy to reproduce and build upon.
Critical weaknesses
-
The cost motivation is not supported. The paper motivates compact backbones by the expense of frontier models, yet it never compares against a frontier model in accuracy or end-to-end cost. Its notion of “compact” is subjective: GPT-5 mini is a paid proprietary API, and Qwen3.5-27B requires high-end GPUs. Moreover, the TTC module relies on GPT-5 itself, together with Gemini-3-Flash and Qwen3.6-Plus, directly contradicting the paper’s premise. This is consequential to my assessment because the paper’s central motivation, efficient long-video understanding without frontier models, is neither validated nor consistently followed.
-
The gains of the pipeline are not disentangled from the backbone. Because the framework mainly combines existing ideas (hierarchical captioning, entity linking, and recursive calls), its value rests on showing that the pipeline itself drives the gains. However, most agentic baselines use older or weaker backbones, and direct inference with Qwen3.5-27B alone already matches or exceeds many of them. In addition, RLVU always builds its memory with Qwen3.5-27B, which strongly affects accuracy, while the memory-construction model of the baselines is not matched. This is the main reason for my rating: without backbone-matched comparisons, it is unclear whether RLVU’s advantage comes from the proposed pipeline or simply from stronger models.
Questions for authors
-
Can you re-run representative agentic baselines with the same reasoning backbones (GPT-5 mini and Qwen3.5-27B) and the same memory-construction VLM, to separate pipeline gains from backbone gains?
-
How does RLVU compare with direct inference using a frontier model (e.g., GPT-5), and with memory-based baselines such as DVD, in both accuracy and end-to-end cost (offline memory construction plus online inference)? What criterion makes GPT-5 mini and Qwen3.5-27B “compact”?
8 comments
W2: The TTC module relies on GPT-5 itself, together with Gemini-3-Flash and Qwen3.6-Plus, directly contradicting the paper’s premise. This is consequential to my assessment because the paper’s central motivation, efficient long-video understanding without frontier models, is neither validated nor consistently followed.
Response: Thank you for raising this concern. We would like to clarify that TTC does not contradict our motivation of reducing reliance on frontier models, for two reasons: (1) Main results are independent of TTC: All primary accuracy and efficiency results are obtained using compact reasoning backbones, without invoking additional frontier models. (2) TTC is disabled by default and serves only as an optional framework extension: It explores how RLVU’s recursive execution can selectively incorporate stronger models for a limited number of difficult, evidence-grounded subproblems, rather than relying on frontier models throughout the entire long-video reasoning process. We explain these points below.
(1) Main results are obtained without TTC. As stated in Secs. 4.1–4.2 and Appendix A.2, all primary accuracy comparisons and online-token analyses use base RLVU with GPT-5 mini or Qwen3.5-27B, without invoking GPT-5, Gemini-3-Flash, or Qwen3.6-Plus. Specifically, without TTC, RLVU achieves 75.9%/71.7%/65.7% with GPT-5 mini and 73.4%/73.1%/68.8% with Qwen3.5-27B on LongVideoBench, LVBench, and Video-MME, respectively. These results demonstrate that RLVU’s main performance gains do not depend on frontier models.
(2) TTC is an optional extension enabled by recursive execution. RLVU’s structured memory and recursive reasoning decompose complex questions into focused subproblems with localized evidence. This provides an opportunity to introduce stronger models only when compact models encounter particularly difficult cases. Rather than using frontier models to process the entire video or handle every reasoning step, TTC focuses their assistance on selected subproblems, while compact models remain responsible for the main reasoning process. This allows RLVU to explore additional reasoning capability without making frontier models a requirement of the framework.
(3) TTC is separate from our primary efficiency claims. TTC explores the potential accuracy improvements available under an additional inference budget. Since stronger-model calls incur extra costs, TTC is evaluated and reported separately from base RLVU. Its results are not included in our primary accuracy or efficiency comparisons, and we do not claim that TTC itself achieves the same cost efficiency as base RLVU.
Overall, RLVU achieves its main results entirely with compact models, while TTC demonstrates the flexibility of its recursive framework to incorporate stronger models selectively when additional resources are available. We will make the default-disabled and optional nature of TTC more explicit in the revised manuscript.
W3: The gains of the pipeline are not disentangled from the backbone. Because the framework mainly combines existing ideas (hierarchical captioning, entity linking, and recursive calls), its value rests on showing that the pipeline itself drives the gains. However, most agentic baselines use older or weaker backbones, and direct inference with Qwen3.5-27B alone already matches or exceeds many of them.
Response: Thank you for raising this concern. RLVU’s improvements are not simply due to stronger backbones, as shown by our new matched-backbone experiments and controlled ablations. We provide three pieces of evidence: (1) new comparisons using the same memory-construction and reasoning models, (2) controlled ablations of memory construction and recursive reasoning, and (3) the methodological differences between RLVU and existing approaches.
(1) New Matched-backbone comparisons. To directly address this concern, we additionally compare RLVU with DVD on the same LVBench subset (20 videos, 266 questions), using Qwen3.5-27B for memory construction and either GPT-5 mini or Qwen3.5-27B for reasoning. Both methods retain their respective memory construction and reasoning procedures. Under these matched settings, RLVU outperforms DVD by 3.7 pp (69.5% vs. 65.8%) with GPT-5 mini and 2.7 pp (72.6% vs. 69.9%) with Qwen3.5-27B. These results show that the improvements cannot be explained solely by stronger models. Detailed results are provided in our response to Q1.
(2) Controlled ablations of memory and reasoning. Tables 2–3 further demonstrate that both memory construction and recursive reasoning contribute to RLVU’s accuracy under controlled settings, rather than relying solely on backbone strength.
- Memory construction (Table 2): With the backbone held fixed, parent-conditioned temporal refinement improves accuracy by 3.7/1.1 pp, while global entity consolidation improves accuracy by 3.8/2.3 pp on LongVideoBench/LVBench.S. These results support the effectiveness of context-aware memory construction and cross-clip entity association.
- Recursive reasoning (Table 3): With identical preconstructed memory and GPT-5 mini backbone, recursive reasoning improves accuracy over non-recursive iterative retrieval by 6.8 pp (69.1% → 75.9%) on LongVideoBench and 4.4 pp (65.7% → 70.1%) on LVBench.S. These gains cannot be attributed to differences in memory or backbone selection.
(3) Methodological distinction. RLVU does not simply combine hierarchical captioning, entity linking, and recursive calls. Instead, it coordinates memory construction with recursive reasoning through a global-to-local-to-global design. Top-down temporal refinement creates multi-scale video descriptions that help reasoning subcalls locate relevant evidence, while bottom-up entity consolidation connects observations across distant clips for cross-event reasoning. Recursive reasoning then retrieves, verifies, and integrates this structured evidence through focused subcalls. Unlike methods that design memory and reasoning separately, RLVU aligns how evidence is organized with how it is accessed during reasoning, reducing both evidence-search and reasoning complexity for compact models.
Together, the matched-backbone comparisons and controlled ablations demonstrate that RLVU’s gains come from its framework design rather than backbone strength alone.
Response: Thank you for raising this concern. Our motivation is to make accurate long-video understanding more affordable and accessible by reducing reliance on expensive frontier models. We observe that many recent long-video understanding methods rely on powerful frontier models, such as GPT-5 and Gemini Pro series, to achieve strong reasoning performance. While effective, repeatedly invoking these models can be costly, particularly for popular online videos, educational lectures, surveillance archives, and recorded meetings that may serve many queries. Compact models offer a more economical alternative but often struggle with evidence localization and complex reasoning across long videos. RLVU addresses this gap through coordinated memory construction and recursive reasoning. We clarify our model-selection criteria and provide additional evidence below.
(1) Lower inference cost. We use “compact” in a relative, cost-oriented sense rather than imposing a fixed parameter-count threshold. GPT-5 mini costs \0.25/2.00 per million input/output tokens, compared with \1.25/10.00 for GPT-5, representing an 80% reduction in both token prices [1][2]. Although GPT-5 mini is a proprietary API, its lower pricing makes it a more affordable alternative to frontier models for repeated inference.
(2) More accessible deployment. Qwen3.5-27B is an open-weight model that supports 4-bit quantization [3]. Under suitable context-length and concurrency settings, its quantized weights can fit within 24 GB of GPU memory, making deployment on consumer GPUs such as the RTX 3090/4090 feasible [4]. Our use of H200 GPUs primarily accelerates experiments rather than reflecting a strict hardware requirement. Practical deployment remains subject to KV-cache and runtime memory requirements.
(3) Additional quantization experiment. To further evaluate efficiency, we compare BF16 and 4-bit Qwen3.5-27B for offline memory construction on 20 LVBench videos (21.18 hours, 266 questions), using the same H200 environment and unchanged online reasoning settings.
Table: Effect of 4-bit Quantization on Memory Construction Efficiency and Accuracy
Quantization reduces offline GPU time by 23.7%, saving 24.6 GPU-minutes per video, with only a 0.75-percentage-point accuracy decrease (193/266 vs. 191/266). These results show that lower-precision memory construction can improve efficiency while largely preserving downstream accuracy.
(4) Comparison with frontier models. To directly address the cost motivation, we additionally compare RLVU with direct GPT-5 inference and DVD in both accuracy and amortized end-to-end cost (Q2). Despite using GPT-5 mini, base RLVU outperforms direct GPT-5 by 11.4 and 11.6 percentage points on LongVideoBench and LVBench, respectively, while trailing by 2.2 points on Video-MME. Our cost analysis further shows that reusable memory can reduce amortized inference costs in repeated-query settings.
Together, these results support our use of cost-efficient reasoning backbones and clarify that “compact” refers to relative inference cost and deployment accessibility, rather than an absolute model-size threshold.
References
[1] OpenAI. “GPT-5 mini.” OpenAI API Documentation. [https://developers.openai.com/api/docs/models/gpt-5-mini]
[2] OpenAI. “GPT-5.” OpenAI API Documentation. [https://developers.openai.com/api/docs/models/gpt-5]
[3] Qwen Team. “Qwen3.5-27B.” Hugging Face Model Card, 2026. [https://huggingface.co/Qwen/Qwen3.5-27B]
[4] Unsloth. “Qwen3.5: How to Run Locally.” Unsloth Documentation, 2026. [https://unsloth.ai/docs/models/qwen3.5]