- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces VideoPursuit, a hierarchical retrieval framework for long-video question answering. It addresses the problem that returning to global reasoning after every retrieval action can interrupt ongoing local exploration. The retrieval agent instead continues gathering information toward a fixed local objective, using a local audit to decide whether further retrieval is needed. It then returns observations and their sources to global reasoning while excluding local plans and tentative conclusions. The framework combines video frames, speech transcripts, and OpenCLIP-based candidate segment navigation to gather evidence and generate answers within a shared action budget. Using Qwen3.8-27B, it achieves an average accuracy of 82.0% across four long-video QA benchmarks. Ablation results also support the effectiveness of persistent local retrieval and the separation of local and global information.
Critical strengths
1.The paper addresses a concrete and practically relevant problem: a single retrieval action often provides a useful lead without resolving the information gap that prompted it, and returning immediately to global reasoning can interrupt further exploration. The framework keeps retrieval focused on the same local objective, uses local audits to determine whether more information is needed, and controls the handoff to prevent local plans and tentative conclusions from entering global reasoning. The design is clear, and its components work toward a common goal.
2.The paper evaluates the method on LVBench, LongVideoBench, Video-MME Long, and MLVU, achieving an average accuracy of 82.0%. This is 10.4 percentage points higher than direct inference with the same backbone and also exceeds the average results reported for AVP and Symphony in the table. The ablations show that persistent local retrieval outperforms returning to global reasoning after every action. They also show that, with persistent local retrieval retained, additionally exposing the full global context or returning the local trajectory reduces accuracy. These results provide reasonably convincing support for the core design choices.
Critical weaknesses
1.The distinction from prior work needs to be made clearer. VideoExplorer, REVEAL, and VideoSEAL have already explored hierarchical retrieval, evidence verification, and the separation of planning from answering. This paper places further emphasis on persistent local retrieval and controlled information handoff, but its ablations mainly compare workflow variants constructed by the authors. These comparisons do not yet establish what capabilities the proposed design adds beyond existing frameworks. The main-table comparisons also involve different backbones and input settings, so a direct comparison supporting the claimed methodological distinction is still missing.
2.The full inference cost has not been adequately evaluated. The appendix reports that the full method uses an average of 3.7 acquisition actions, compared with 4.6 for Action-Level Return, showing that its gains do not come from taking more retrieval actions. However, acquisition counts do not capture the cost of local audits, global assessments, or context processing. Fewer actions therefore do not necessarily imply lower latency or token usage. Without complete model-call and timing statistics, it remains difficult to assess the trade-off between the accuracy gains and the actual computational cost.
Questions for authors
- Could the authors report the average input and output token counts, number of model calls, and end-to-end latency in seconds per question for VideoPursuit, Action-Level Return, and direct inference? In particular, does the reduction in acquisition actions offset the cost of the additional local audits? Please specify the hardware and concurrency settings, and distinguish the cost of initial indexing and ASR preprocessing from per-question inference with cached preprocessing results.
2.For questions that depend on event order or evidence from multiple segments, do the candidate intervals provided by OpenCLIP cover the necessary evidence? The appendix shows that local retrieval can correct an early localization error and resolve temporal order, but it is unclear how consistently this recovery works. Could the authors report candidate evidence coverage, the additional acquisition actions needed after navigation errors, and the proportion of runs that exhaust the budget? Comparisons that remove the index or replace the navigation features would also help clarify how candidate navigation affects accuracy and search efficiency.
3.What fundamentally distinguishes VideoPursuit from existing retrieval-agent methods such as VideoExplorer, REVEAL, and VideoSEAL in its retrieval control and information flow? Do these differences go beyond workflow changes such as organizing multiple retrieval actions into a local loop or adjusting which context each component can access? A concrete comparison of the mechanisms, or controlled experiments, would help clarify which capabilities are introduced by this design and are not already available in existing methods.
4 comments
Author Response to Reviewer Um7z [2/2]
W2/Q1: Computational Costs and Preprocessing Overhead
In particular, does the reduction in acquisition actions offset the cost of the additional local audits?
Net Cost of Acquisition Actions and Local Audits. Compared with Action-Level Return (A1), Full reduces the average number of visual acquisition actions from 4.6 to 3.7, total token consumption from 179,925 to 114,990, and online inference latency from 131.2 to 89.6 seconds. Even after accounting for Local Audit overhead, Full substantially reduces both total token consumption and online latency, demonstrating that the overall savings outweigh the additional auditing costs. Complete input/output token counts, model call counts, end-to-end latency, hardware and concurrency settings, and offline preprocessing cost analyses are provided in Appendix B.5 of the revised manuscript.
Q2: Candidate Evidence Coverage and Recovery from Navigation Errors
Could the authors report candidate evidence coverage, the additional acquisition actions needed after navigation errors, and the proportion of runs that exhaust the budget?
We analyze 715 LongVideoBench questions with multiple annotated temporal reference locations to evaluate evidence acquisition across multiple video segments. Among these, we select 192 questions involving at least two Pursuits and multiple rounds of local retrieval to examine initial candidate coverage and subsequent recovery. We calculate the acquisition-budget exhaustion rate across all 715 questions.
- Initial Candidate Coverage: The initial OpenCLIP Top-5 candidates cover at least one official reference location for 50.0% (96/192) of questions and all reference locations for 33.3% (64/192).
- Subsequent Recovery: Among the 128 questions with incomplete initial candidate coverage, 35.2% (45/128) achieve complete coverage through subsequent visual observations. Among the 146 questions whose first actual visual observations still provide incomplete coverage, 32.9% (48/146) recover complete coverage through additional acquisitions. Successful recovery requires an average of 2.25 additional visual acquisitions.
- Budget Exhaustion: Among all 715 questions, 3.9% (28/715) exhaust the acquisition budget, after which the system produces an answer based on the evidence already acquired.
Additional experimental details are provided in Appendix B.8 of the revised manuscript.
Kind Follow-up on Our Rebuttal Submission
Dear Reviewer Um7z,
Thank you again for your thoughtful review.
We have provided detailed responses to your concerns and updated our manuscript with additional analyses of computational costs, navigation recovery, and comparisons with existing retrieval mechanisms.
As the discussion period is approaching its end, we would greatly appreciate it if you could take a look at our responses and let us know whether any concerns remain.
Thank you for your time and consideration of our responses in your final evaluation.
Best regards, The Authors
Thank you for your response. After considering your points, I have decided to keep my original score.
Author Response to Reviewer Um7z [1/2]
We thank the reviewer for recognizing the practical significance of the research problem, the designs of persistent local retrieval and selective information delivery, and the supporting experimental results. In the revised manuscript, we add a matched-backbone AVP-style comparison in Section 4.3 (Table 2) and the Need-Locked control in Section 4.3.1 (Figure 3), with detailed ablation settings and results in Appendices B.2–B.4. The fixed-trajectory replay experiment in Appendix B.6 further examines selective information delivery. We also report computational costs, online latency, and indexing overhead in Appendix B.5, and analyze candidate coverage, recovery through additional acquisitions, and the frequency of reaching the acquisition limit in Appendix B.8. All revisions are highlighted in blue.
W1/Q3: Fundamental Differences from Existing Retrieval Agents
1. Retrieval Control Architecture
The core innovation of VideoPursuit is to redefine the control granularity of Agentic Video Retrieval, transforming Retrieval Actions driven by Global Reasoning into Retrieval Processes organized around individual Search Needs and continuously driven by Local Reasoning.
Existing Agentic Video Retrieval methods enable dynamic exploration and evidence verification, but differ in how they organize retrieval control:
These methods still primarily rely on Global Reasoning to coordinate retrieval, rather than establishing independent evidence acquisition processes with persistent working states, autonomous continuation decisions, and explicit termination criteria.
VideoPursuit instead introduces Need-Level Retrieval Control between Global Reasoning and retrieval actions. The Global Controller proposes Search Needs, while the Retrieval Agent continuously acquires evidence for each Need based on accumulated observations and unresolved gaps. Local Audit evaluates evidence sufficiency and guides continuation or repair, while Observation Projection selectively delivers acquired evidence to Global Reasoning.
This shifts the fundamental control unit of retrieval from individual actions to persistent, Need-Scoped Local Retrieval Processes, enabling each Search Need to be pursued through continuous local reasoning until its evidence objective is satisfied.
2. Mechanism Validation
We further evaluate the proposed retrieval control and information flow mechanisms through controlled ablations and a matched-backbone agent comparison.
On 597 questions, Action-Level Local Status Feedback achieves 68.7% accuracy. Preserving the current Search Need through Need-Locked improves accuracy to 73.5%, while the complete Persistent Local Retrieval further reaches 81.1%, validating the contributions of objective persistence and local process control.
With existing retrieval trajectories and observations fixed, selective Observation Projection outperforms complete local-trajectory delivery by 5.1 percentage points supporting the design of the local–global information boundary.
Furthermore, under a unified backbone model and retrieval configuration, VideoPursuit outperforms an AVP-style global action-level retrieval loop by 6.7 percentage points (81.1% vs. 74.4%).
The matched-backbone AVP-style comparison is provided in §4.3 (Table 2) of the revised manuscript. Retrieval control settings, complete ablations, and Need-Locked analysis are presented in Appendices B.2–B.4, while fixed-trajectory replay is detailed in Appendix B.6.