acceptodds
Under review as a conference paper at ICLR 2027

VideoPursuit: From Retrieval Actions to Local Retrieval Processes for Agentic Long-Video Understanding

12 views

Abstract

Agentic long-video understanding increasingly relies on adaptive video retrieval to access sparse, task-relevant observations during reasoning. However, many existing methods follow an action-level handoff pattern, returning control to global reasoning after each retrieval action. A useful retrieval action may still only narrow a temporal region, reveal a new cue, or indicate what should be inspected next, so completing the action does not necessarily mean that the current retrieval is ready for handoff. We introduce VideoPursuit, a framework that treats retrieval for each local objective as a dedicated process rather than a sequence of retrieval actions individually managed by global reasoning. When global reasoning over the original question identifies an information gap, it delegates a local retrieval objective to a dedicated retrieval agent. Within a bounded local execution scope, the agent may perform multiple retrieval actions around the same objective, using explicit local audit after each action to determine whether further retrieval is needed or the current result is ready for handoff. Intermediate retrieval states remain local, while acquired observations and their source information are selectively returned to global reasoning. Across four long-video understanding benchmarks, VideoPursuit achieves an average accuracy of 82.0%, improving by 3.4 percentage points over the strongest prior agentic method. Ablations further show that persistent local retrieval substantially outperforms action-level return, while selective information propagation across the local-global boundary provides additional gains.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
long-video understanding, agentic video understanding, video reasoning, video-language models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Um7zOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces VideoPursuit, a hierarchical retrieval framework for long-video question answering. It addresses the problem that returning to global reasoning after every retrieval action can interrupt ongoing local exploration. The retrieval agent instead continues gathering information toward a fixed local objective, using a local audit to decide whether further retrieval is needed. It then returns observations and their sources to global reasoning while excluding local plans and tentative conclusions. The framework combines video frames, speech transcripts, and OpenCLIP-based candidate segment navigation to gather evidence and generate answers within a shared action budget. Using Qwen3.8-27B, it achieves an average accuracy of 82.0% across four long-video QA benchmarks. Ablation results also support the effectiveness of persistent local retrieval and the separation of local and global information.

Critical strengths

1.The paper addresses a concrete and practically relevant problem: a single retrieval action often provides a useful lead without resolving the information gap that prompted it, and returning immediately to global reasoning can interrupt further exploration. The framework keeps retrieval focused on the same local objective, uses local audits to determine whether more information is needed, and controls the handoff to prevent local plans and tentative conclusions from entering global reasoning. The design is clear, and its components work toward a common goal.

2.The paper evaluates the method on LVBench, LongVideoBench, Video-MME Long, and MLVU, achieving an average accuracy of 82.0%. This is 10.4 percentage points higher than direct inference with the same backbone and also exceeds the average results reported for AVP and Symphony in the table. The ablations show that persistent local retrieval outperforms returning to global reasoning after every action. They also show that, with persistent local retrieval retained, additionally exposing the full global context or returning the local trajectory reduces accuracy. These results provide reasonably convincing support for the core design choices.

Critical weaknesses

1.The distinction from prior work needs to be made clearer. VideoExplorer, REVEAL, and VideoSEAL have already explored hierarchical retrieval, evidence verification, and the separation of planning from answering. This paper places further emphasis on persistent local retrieval and controlled information handoff, but its ablations mainly compare workflow variants constructed by the authors. These comparisons do not yet establish what capabilities the proposed design adds beyond existing frameworks. The main-table comparisons also involve different backbones and input settings, so a direct comparison supporting the claimed methodological distinction is still missing.

2.The full inference cost has not been adequately evaluated. The appendix reports that the full method uses an average of 3.7 acquisition actions, compared with 4.6 for Action-Level Return, showing that its gains do not come from taking more retrieval actions. However, acquisition counts do not capture the cost of local audits, global assessments, or context processing. Fewer actions therefore do not necessarily imply lower latency or token usage. Without complete model-call and timing statistics, it remains difficult to assess the trade-off between the accuracy gains and the actual computational cost.

Questions for authors
  1. Could the authors report the average input and output token counts, number of model calls, and end-to-end latency in seconds per question for VideoPursuit, Action-Level Return, and direct inference? In particular, does the reduction in acquisition actions offset the cost of the additional local audits? Please specify the hardware and concurrency settings, and distinguish the cost of initial indexing and ASR preprocessing from per-question inference with cached preprocessing results.

2.For questions that depend on event order or evidence from multiple segments, do the candidate intervals provided by OpenCLIP cover the necessary evidence? The appendix shows that local retrieval can correct an early localization error and resolve temporal order, but it is unclear how consistently this recovery works. Could the authors report candidate evidence coverage, the additional acquisition actions needed after navigation errors, and the proportion of runs that exhaust the budget? Comparisons that remove the index or replace the navigation features would also help clarify how candidate navigation affects accuracy and search efficiency.

3.What fundamentally distinguishes VideoPursuit from existing retrieval-agent methods such as VideoExplorer, REVEAL, and VideoSEAL in its retrieval control and information flow? Do these differences go beyond workflow changes such as organizing multiple retrieval actions into a local loop or adjusting which context each component can access? A concrete comparison of the mechanisms, or controlled experiments, would help clarify which capabilities are introduced by this design and are not already available in existing methods.

4 comments
AuthorsOpenReview ↗

Author Response to Reviewer Um7z [1/2]

We thank the reviewer for recognizing the practical significance of the research problem, the designs of persistent local retrieval and selective information delivery, and the supporting experimental results. In the revised manuscript, we add a matched-backbone AVP-style comparison in Section 4.3 (Table 2) and the Need-Locked control in Section 4.3.1 (Figure 3), with detailed ablation settings and results in Appendices B.2–B.4. The fixed-trajectory replay experiment in Appendix B.6 further examines selective information delivery. We also report computational costs, online latency, and indexing overhead in Appendix B.5, and analyze candidate coverage, recovery through additional acquisitions, and the frequency of reaching the acquisition limit in Appendix B.8. All revisions are highlighted in blue.

W1/Q3: Fundamental Differences from Existing Retrieval Agents

What fundamentally distinguishes VideoPursuit from existing retrieval agents ?

1. Retrieval Control Architecture

The core innovation of VideoPursuit is to redefine the control granularity of Agentic Video Retrieval, transforming Retrieval Actions driven by Global Reasoning into Retrieval Processes organized around individual Search Needs and continuously driven by Local Reasoning.

Existing Agentic Video Retrieval methods enable dynamic exploration and evidence verification, but differ in how they organize retrieval control:

  • VideoExplorer delegates temporal grounding and relevance verification to a Grounder, while subsequent exploration remains coordinated by the Planner.
  • REVEAL uses Rubric-Guided Verification to identify evidence gaps, but subsequent retrieval remains driven by iterative Global Reasoning.
  • VideoSEAL separates evidence assessment from retrieval planning through an Inspector–Planner architecture, while subsequent retrieval decisions remain with the Planner.

These methods still primarily rely on Global Reasoning to coordinate retrieval, rather than establishing independent evidence acquisition processes with persistent working states, autonomous continuation decisions, and explicit termination criteria.

VideoPursuit instead introduces Need-Level Retrieval Control between Global Reasoning and retrieval actions. The Global Controller proposes Search Needs, while the Retrieval Agent continuously acquires evidence for each Need based on accumulated observations and unresolved gaps. Local Audit evaluates evidence sufficiency and guides continuation or repair, while Observation Projection selectively delivers acquired evidence to Global Reasoning.

This shifts the fundamental control unit of retrieval from individual actions to persistent, Need-Scoped Local Retrieval Processes, enabling each Search Need to be pursued through continuous local reasoning until its evidence objective is satisfied.

2. Mechanism Validation

We further evaluate the proposed retrieval control and information flow mechanisms through controlled ablations and a matched-backbone agent comparison.

On 597 questions, Action-Level Local Status Feedback achieves 68.7% accuracy. Preserving the current Search Need through Need-Locked improves accuracy to 73.5%, while the complete Persistent Local Retrieval further reaches 81.1%, validating the contributions of objective persistence and local process control.

With existing retrieval trajectories and observations fixed, selective Observation Projection outperforms complete local-trajectory delivery by 5.1 percentage points supporting the design of the local–global information boundary.

Furthermore, under a unified backbone model and retrieval configuration, VideoPursuit outperforms an AVP-style global action-level retrieval loop by 6.7 percentage points (81.1% vs. 74.4%).

The matched-backbone AVP-style comparison is provided in §4.3 (Table 2) of the revised manuscript. Retrieval control settings, complete ablations, and Need-Locked analysis are presented in Appendices B.2–B.4, while fixed-trajectory replay is detailed in Appendix B.6.

AuthorsOpenReview ↗

Author Response to Reviewer Um7z [2/2]

W2/Q1: Computational Costs and Preprocessing Overhead

In particular, does the reduction in acquisition actions offset the cost of the additional local audits?

Net Cost of Acquisition Actions and Local Audits. Compared with Action-Level Return (A1), Full reduces the average number of visual acquisition actions from 4.6 to 3.7, total token consumption from 179,925 to 114,990, and online inference latency from 131.2 to 89.6 seconds. Even after accounting for Local Audit overhead, Full substantially reduces both total token consumption and online latency, demonstrating that the overall savings outweigh the additional auditing costs. Complete input/output token counts, model call counts, end-to-end latency, hardware and concurrency settings, and offline preprocessing cost analyses are provided in Appendix B.5 of the revised manuscript.

Q2: Candidate Evidence Coverage and Recovery from Navigation Errors

Could the authors report candidate evidence coverage, the additional acquisition actions needed after navigation errors, and the proportion of runs that exhaust the budget?

We analyze 715 LongVideoBench questions with multiple annotated temporal reference locations to evaluate evidence acquisition across multiple video segments. Among these, we select 192 questions involving at least two Pursuits and multiple rounds of local retrieval to examine initial candidate coverage and subsequent recovery. We calculate the acquisition-budget exhaustion rate across all 715 questions.

  1. Initial Candidate Coverage: The initial OpenCLIP Top-5 candidates cover at least one official reference location for 50.0% (96/192) of questions and all reference locations for 33.3% (64/192).
  2. Subsequent Recovery: Among the 128 questions with incomplete initial candidate coverage, 35.2% (45/128) achieve complete coverage through subsequent visual observations. Among the 146 questions whose first actual visual observations still provide incomplete coverage, 32.9% (48/146) recover complete coverage through additional acquisitions. Successful recovery requires an average of 2.25 additional visual acquisitions.
  3. Budget Exhaustion: Among all 715 questions, 3.9% (28/715) exhaust the acquisition budget, after which the system produces an answer based on the evidence already acquired.

Additional experimental details are provided in Appendix B.8 of the revised manuscript.

AuthorsOpenReview ↗

Kind Follow-up on Our Rebuttal Submission

Dear Reviewer Um7z,

Thank you again for your thoughtful review.

We have provided detailed responses to your concerns and updated our manuscript with additional analyses of computational costs, navigation recovery, and comparisons with existing retrieval mechanisms.

As the discussion period is approaching its end, we would greatly appreciate it if you could take a look at our responses and let us know whether any concerns remain.

Thank you for your time and consideration of our responses in your final evaluation.

Best regards, The Authors

Thank you for your response. After considering your points, I have decided to keep my original score.

Reviewer_iM8cOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Summary This paper studies the granularity of retrieval control in agentic long-video understanding: existing methods return control to global reasoning after each retrieval action, but a single action often only narrows a range and leaves the local objective unresolved. VideoPursuit organizes retrieval for each local objective as a persistent process, where a retrieval agent runs multiple actions within a bounded scope, uses a local audit to decide when to hand off, selectively returns only observations, and separates local retrieval adequacy from global answer sufficiency. With a Qwen3.8-27B backbone it reaches 82.0% average across four benchmarks. Contributions

  1. It identifies a reasonable control gap: judging retrieval against the original question makes it hard to tell whether the local objective is resolved.
  2. It proposes VideoPursuit, delegating each local objective to a dedicated agent that retrieves persistently, decides handoff locally, and returns only observations, positioned as a framework-level contribution for long-video retrieval.
  3. It validates on four benchmarks and ablates persistent local retrieval and the information boundary. Validation The method averages 82.0% across four benchmarks, 10.4 points over same-backbone direct inference and 3.4 over AVP. Ablations on 597 questions show persistent local retrieval (81.1%) clearly beats Action Return (64.0%) and Terminate Status (68.7%) while using fewer actions, and the selective information boundary outperforms full two-way sharing, which is fairly convincing. The main limitation is that the primary comparison is cross-backbone and system-level, so mechanism attribution is not clean, and there is no fair comparison against the latest native long-context models.
Critical strengths
  1. The core insight is well-motivated: it explicitly separates local retrieval adequacy from global answer sufficiency, a sound design principle for long-video tasks that must first locate an event before comparing relations.
  2. The central ablation is fairly convincing: persistent local retrieval (81.1%) clearly beats Action Return (64.0%) and Terminate Status (68.7%) while using fewer retrieval actions, so the gain is not from simply retrieving more.
  3. Implementation and reproducibility material is reasonably complete: the appendix details the temporal index construction, retrieval action types, budgets, and each agent's inputs/outputs, with two timestamped case trajectories.
Critical weaknesses
  1. The main comparison is cross-system, so gains cannot be attributed to the mechanism. VideoPursuit uses Qwen3.8-27B while the strongest baseline AVP uses Gemini-2.5-Pro, with different backbones, tools, and budgets; Direct Inference is not a fair agentic baseline either. A ReAct/AVP-style loop under the same backbone, tools, and budget is needed.
  2. The core ablation does not isolate individual mechanisms. The 12.4-point gain over Terminate Status mixes objective holding, local-state preservation, interrupting global reasoning, and context isolation, and the information-boundary drops are uncontrolled for context length and evidence, which the paper itself concedes. Controlled variants are missing.
  3. End-to-end cost is absent and reproduction is hard. The 204 frames exclude OpenCLIP indexing, Controller calls, tokens, and latency; the backbone is cited as Qwen3.8-Max but called Qwen3.8-27B, and prompts are abridged. Report calls, tokens, latency, and indexing cost, and recheck the 81.1% under matched budget.
Questions for authors
  1. Under the same Qwen3.8-27B backbone, retrieval tools, and compute budget, if an AVP/ReAct-style global retrieval loop is reimplemented and compared head-to-head, does VideoPursuit still outperform it? Can the current 3.4-point cross-system gap be read as mechanism-level causal evidence?
  2. Can you add a controlled variant that still returns to global reasoning each step but forces the same fixed local objective, to separate the benefit of persistent local retrieval from objective holding? What are the relative contributions within the 12.4-point gain?
  3. In the information-boundary study, when observations and token counts are exactly matched and only the exposure of the Local Audit conclusion is changed, do the 4.0/5.4-point gains persist? How do you rule out confounds from context length and evidence differences, given the paper concedes the two runs differ in their sampled observations?
  4. Can you report end-to-end cost: average calls per question, each agent's input/output tokens, total latency, and the OpenCLIP indexing overhead and its reuse count? Does the 81.1% ablation advantage hold under matched compute budget? Also, please clarify whether the backbone is Qwen3.8-27B or the Qwen3.8-Max in the citation.
  5. As frontier models rapidly improve, both closed (e.g., 6astra) and open (e.g., Qwen3.8-27B) models offer ever stronger native long-context video understanding. Is the video reasoning task this paper targets still a real problem? More importantly, relative to simply plugging such strong models into a general-purpose base harness (e.g., Codex), where does your method's value and irreplaceability lie? Please provide a direct comparison against a "base harness + frontier model" setup.
4 comments
AuthorsOpenReview ↗

Author Response to Reviewer iM8c [1/2]

We thank the reviewer for recognizing the distinction between local retrieval sufficiency and global answer sufficiency, as well as the core ablation experiments. In the revised manuscript, we add a matched-backbone AVP-style comparison in Section 4.3 (Table 2) and the Need-Locked objective-preservation experiment in Section 4.3.1 (Figure 3), with its execution constraints detailed in Appendix B.4. The fixed-trajectory replay experiment appears in Appendix B.6, and token consumption, model-call counts, online latency, and indexing costs are reported in Appendix B.5. Comparisons with the general-purpose agent harnesses Codex and OpenCode are presented in Appendix B.7, reporting their accuracy and token consumption under the same backbone and retrieval tools. All revisions are highlighted in blue.

W1/Q1: Matched-Backbone, Matched-Tool, and Matched-Budget Agent Comparison

Does VideoPursuit still outperform AVP/ReAct-style global retrieval loop?

To address the concern about cross-system comparisons and mechanism attribution, we introduce an AVP-style global retrieval loop baseline using Qwen3.8-27B, evaluated under identical backbone, tool, and budget constraints.

MethodCorrect / TotalAccuracy
AVP-style + Qwen3.8-27B444/59774.4%
VideoPursuit + Qwen3.8-27B484/59781.1%

On the same 597 questions, VideoPursuit outperforms the baseline by 6.7 percentage points. This matched comparison provides direct evidence for the contribution of persistent local retrieval control beyond the original 3.4-point cross-system performance difference.

W2/Q2: Controlled Comparison with a Fixed Local Objective

Add a controlled variant that still returns to global reasoning each step but forces the same fixed local objective

We would like to clarify the reviewer's statement that “Controlled variants are missing.” The original manuscript already presents controlled A1–A4 ablations in §4.3 and Appendix B.2, with the backbone model, initial observations, retrieval capabilities, per-question resource limits, and evaluation question set held constant. The reviewer's request concerns further isolation of individual mechanisms, rather than the absence of controlled experiments in the original manuscript.

To directly separate objective holding from persistent local retrieval, we introduce Need-Locked. Building upon A2, when Global decides to continue retrieval, the execution controller programmatically enforces preservation of the current unresolved Search Need, while Global remains responsible for the continuation decision.

Need-Locked improves accuracy from 68.7% to 73.5% (+4.8 points) over A2, while the complete VideoPursuit further reaches 81.1% (+7.6 points).

These results validate the independent contribution of Objective Persistence and demonstrate that the advantage of the complete Pursuit does not arise solely from objective preservation, but also from the integrated execution mechanism in which local state continuously participates in reasoning and continuation decisions are made locally.

W2/Q3: Information Boundary and Fixed-Trajectory Replay

How do you rule out confounds from context length and evidence differences, given the paper concedes the two runs differ in their sampled observations?

We would like to clarify the statement “uncontrolled for context length and evidence, which the paper itself concedes” in W2. The original manuscript does not concede that the information-boundary experiments lack experimental controls. Appendix B.2 states that different variants may acquire different subsequent observations because they make different retrieval decisions. Such differences are a natural consequence of dynamic Agentic Retrieval and do not imply that the underlying experimental conditions were uncontrolled.

The original ablations compare the complete execution outcomes under different information-boundary designs. Nevertheless, to address the reviewer's more specific request, we further examine whether the advantage persists when the acquired evidence is held fixed. To this end, we introduce Fixed-Trajectory Global Replay. We fix the local retrieval trajectories, observations, and Audit records produced by Full, compare selective Observation Projection against delivery of the complete local trajectory, and rerun Global answer generation.

On the same 597 questions and with the same acquired evidence, selective delivery achieves 81.1% accuracy, compared with 76.0% for complete-trajectory delivery, an improvement of 5.1 percentage points.

These results show that selective information delivery retains a clear advantage even when the existing retrieval trajectories and observations are fixed, further supporting the effectiveness of the local–global information boundary design. Detailed settings are provided in Appendix B.6 of the revised manuscript.

AuthorsOpenReview ↗

Author Response to Reviewer iM8c [2/2]

W3/Q4: Computational Costs and Model Identification

Can you report costs, including model calls, tokens, latency, and OpenCLIP indexing overhead and reuse?

Clarify whether the backbone is Qwen3.8-27B or the Qwen3.8-Max in the citation.

Model Calls and Per-Agent Token Usage. Beyond the reported frame acquisitions, VideoPursuit makes an average of 12.06 model calls per question, including initialization. The average input and output token counts for each agent are reported below.

AgentInput Tokens / QuestionOutput Tokens / Question
Controller (including initialization)81,0242,470
Retrieval Agent25,197436
Local Audit5,739124

The average online QA latency is 89.6 seconds per question. OpenCLIP indexing takes an average of 44.9 seconds per video.The 597 questions cover 465 unique videos, each indexed only once; cross-question reuse avoids 132 repeated index builds.

VideoPursuit uses an average of 114,990 tokens per question, lower than the 179,925 tokens required by Action-Level Return, while achieving higher accuracy. OpenCLIP indices are constructed per unique video and can be reused across multiple questions referring to the same video. Additional configuration details are provided in Appendix B.5 of the revised manuscript.

Model Identification. The model used in this work is Qwen3.8-27B, not Qwen3.8-Max. The title Qwen3.8-Max: A New Bar for Coding and Cowork, mentioned by the reviewer, is the BibTeX citation title provided by the official Hugging Face model card for Qwen3.8-27B, rather than an incorrectly cited reference. The original manuscript follows the official citation format. The reference title should not be confused with the identity of the model actually used in our experiments.

Q5: Comparison with General-Purpose Agent Harnesses and Stronger Models

What is the value of VideoPursuit compared with a general-purpose agent harness (e.g., Codex)?

Long-video understanding depends not only on the reasoning capabilities and context capacity of the underlying model, but also on the selection and acquisition of long-range visual evidence and the associated computational costs. Therefore, specialized video evidence retrieval workflows and general-purpose model capabilities offer complementary value.

We further integrate two general-purpose Agent Harnesses, Codex and OpenCode, with the same Qwen3.8-27B backbone, providing identical video inputs, ASR information, and retrieval tools.

VideoPursuit achieves 81.1% accuracy, outperforming Codex at 78.9% by 2.2 percentage points, while using only 39.5% of its average token budget. Compared with OpenCode at 77.1%, VideoPursuit also achieves higher accuracy while consuming only 26.7% of its average tokens.

These results demonstrate that, under a shared model backbone and retrieval tools, the specialized long-video retrieval control workflow achieves both higher answer accuracy and lower inference token consumption.

The execution settings and complete comparison results for general-purpose Agent Harnesses are provided in Appendix B.7 of the revised manuscript.

AuthorsOpenReview ↗

Kind Follow-up on Our Rebuttal Submission

Dear Reviewer iM8c,

Thank you again for your thoughtful review.

We have provided detailed responses to your concerns and updated our manuscript with additional matched-backbone comparisons, objective-preservation and fixed-trajectory replay experiments, and computational cost analyses.

As the discussion period is approaching its end, we would greatly appreciate it if you could take a look at our responses and let us know whether any concerns remain.

Thank you for your time and consideration of our responses in your final evaluation.

Best regards, The Authors

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer iM8c,

As the discussion period is nearing its end, we would like to kindly follow up on our responses and revised manuscript. We would be grateful if you could take another look at the additional evidence and let us know whether it addresses your concerns.

Thank you for your time and consideration.

Reviewer_zD62OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes VideoPursuit, an agentic framework for long-video question answering. Its central idea is to treat retrieval for a local information need as a persistent local process, rather than returning control to global reasoning after every retrieval action. A controller identifies an information gap and delegates a neutral retrieval objective to a retrieval agent. The local agent can perform several acquisitions around that objective, while a local audit decides whether to deliver the observations, repair the retrieval, or report a limitation. Only selected observations and source information are returned to global reasoning. Using Qwen3.8-27B, offline WhisperX transcripts, an initial 128-frame overview, and an OpenCLIP temporal index, the method is evaluated on LVBench, LongVideoBench, Video-MME Long, and MLVU. It reports an average accuracy of 82.0%. On a 597-question ablation subset, persistent local retrieval achieves 81.1%, compared with 68.7% when local status is reported but control returns after every action. Restricting information propagation across the local-global boundary also outperforms exposing either the full global context locally or the full local trace globally.

Critical strengths
  1. The paper isolates a clear and practically relevant agent-design question. The comparison between immediate action return, return with local-status feedback, and persistent local execution directly tests whether a local objective should retain control across multiple acquisitions. The 12.4-point difference between status feedback and persistent retrieval is substantial and cannot be explained by a larger acquisition count, since the full method actually uses fewer acquisition actions on average.
  2. The local-global information interface is carefully designed and documented. The controller, retrieval agent, and audit have distinct responsibilities, and the appendix specifies their inputs, outputs, stopping rules, action ceilings, and prompts. The information-boundary ablations and qualitative examples support the claim that selectively returning observations can prevent provisional local interpretations from contaminating global reasoning.
Critical weaknesses
  1. The headline comparison mixes substantially different models, inputs, and evaluation pipelines. Published baselines use different reasoning backbones, visual models, transcript availability, frame budgets, and agent-call budgets. The same-backbone direct Qwen baseline is valuable, but it does not distinguish the VideoPursuit control structure from a strong alternative agent using the same backbone, tools, prompts, and total inference budget. The 3.4-point claimed improvement over the leading published agent therefore should not be interpreted as a clean method-level comparison.
  2. The ablations establish that the authors’ implementation of persistent local retrieval substantially outperforms action-level return, but they provide limited evidence about the mechanism behind this unusually large gain. Persistent execution simultaneously changes state continuity, objective stability, context exposure, prompt structure, and the locus of control. The paper does not disentangle these factors or quantify how often later local acquisitions repair a specific gap identified by the audit. As a result, the experiments demonstrate an implementation-level effect but do not yet establish the proposed process-level explanation.
Questions for authors
  1. Can the authors report per-benchmark ablations, paired per-question outcome changes, confidence intervals or paired significance tests, and the random seed/question roster for the 597-question subset? Are the conclusions stable across multiple subsets or decoding runs?
  2. Can the authors disentangle why persistent local retrieval improves accuracy by 12.4 points over local-status feedback? In particular, please separately control for preservation of the local trajectory, reuse of the exact same objective, context length, prompt/token budget, and the location of the continuation decision. It would also be valuable to report how often a second or later local acquisition resolves the gap identified by the preceding audit, changes the final answer, or causes harmful local fixation.
4 comments
AuthorsOpenReview ↗

Author Response to Reviewer zD62 [1/2]

We thank the reviewer for recognizing the motivation behind Persistent Local Retrieval, the core ablation results, and the design of the local–global information interface. In the revised manuscript, we add a matched-backbone AVP-style comparison in Section 4.3 (Table 2) and the Need-Locked control in Section 4.3.1 (Figure 3). Appendix B.3 provides per-benchmark results, paired question-level comparisons, and repeated runs with different decoding seeds, while Appendix B.4 details the Need-Locked execution constraints and subsequent retrieval behavior analyses. Appendix B.5 specifies prompt settings for shared agent roles, context and token budgets, and inference costs. The fixed-trajectory replay experiment is presented in Appendix B.6. All revisions are highlighted in blue.

W1: Matched-Backbone Agent Comparison

Does VideoPursuit outperform an AVP/ReAct-style agent under matched conditions?

We introduce an additional AVP-style global retrieval loop baseline using Qwen3.8-27B, evaluated under the same controlled conditions. Detailed experimental settings are provided in Appendix B.5 of the revised manuscript.

MethodCorrect / TotalAccuracy
AVP-style + Qwen3.8-27B444/59774.4%
VideoPursuit + Qwen3.8-27B484/59781.1%

VideoPursuit outperforms the baseline by 6.7 percentage points.

W2/Q2: Mechanistic Explanation and Factor Decomposition of Persistent Local Retrieval

Can the authors disentangle the mechanisms behind the 12.4-point gain and analyze subsequent local retrieval behavior?

The central motivation of VideoPursuit is that completing a Retrieval Action does not necessarily mean that the current Search Need has been resolved. We therefore transform action-by-action retrieval driven by Global Reasoning into a persistent retrieval process driven by Local Reasoning. This allows the Retrieval Agent to continuously acquire evidence for the same Search Need, preserving the continuity of local reasoning state while reducing the context overhead and information interference caused by repeated handoffs to Global Reasoning.

1. Objective Persistence: Reuse of the Exact Same Objective

We introduce a new Need-Locked variant. Building upon A2, Need-Locked programmatically enforces reuse of the exact same unresolved Search Need whenever Global decides to continue retrieval.

Under the same evaluation, A2 achieves 68.7% accuracy, Need-Locked reaches 73.5%(+4.8 points), and Full further improves to 81.1%(+7.6 points). These results support the contribution of objective persistence and demonstrate the additional benefits of the complete persistent local execution mechanism.

2. Local Trajectory Preservation and Location of the Continuation Decision

Action Return already preserves historical observations. The key distinction of Full is not simply the addition of historical information, but allowing this information to continuously participate in reasoning and retrieval decisions within the same Pursuit.

In Need-Locked, each acquisition action still returns control to Global for subsequent decisions. In Full, retrieval is continuously advanced within the local Pursuit.

Thus, the difference between Need-Locked and Full reflects the contribution of the complete local execution mechanism, including local trajectory preservation and the location of the continuation decision.

3. Context Length, Prompt/Token Budget, and Subsequent Retrieval Behavior.

The ablation experiments use a context limit of 100,000 tokens per LLM call and a cumulative budget limit of 200,000 tokens per question. Among the action-level ablation variants, Full achieves the highest accuracy while consuming the fewest total tokens on average. Token costs and budget configurations for each retrieval-control variant are detailed in Appendix B.5 of the revised manuscript.

Subsequent Retrieval Behavior

From the 597 evaluated questions, we identify 132 involving subsequent corrective local acquisitions and analyze their outcomes. A second or later local acquisition resolves the gap identified by the preceding Audit in 53.8% of cases, changes the final answer in 31.1%, and causes harmful local fixation in 11.4%. Definitions and question-level counts are reported in Appendix B.4.

AuthorsOpenReview ↗

Author Response to Reviewer zD62 [2/2]

Q1: Per-Benchmark Ablations, Per-Question Statistics, and Reproducibility

Per-Benchmark Ablation Results (Accuracy, %):

Benchmark (N)FullA1A2A3A4
LVBench (180)75.063.367.876.775.6
LongVideoBench (150)81.358.763.371.370.0
Video-MME Long (140)82.967.972.981.480.0
MLVU (127)87.466.971.779.578.0
Overall (597)81.164.068.777.175.7

A1: Action Return; A2: Terminate Status; A3: Global Context; A4: Local Trace.

Paired Per-Question Outcomes and Confidence Intervals:

BaselineFull OnlyBaseline OnlyΔ (pp)95% CI (pp)
A116967+17.1[12.2, 22.0]
A215177+12.4[7.5, 17.3]
A37248+4.0[0.4, 7.6]
A411987+5.4[0.7, 10.1]

Question List and Random Settings. The question-sampling seed is 42, and the question list is provided in the supplementary materials.

Different Decoding Seeds. The original decoding run uses seed 906, and the additional run uses seed 1003. The additional run achieves 80.9% accuracy.

Per-benchmark ablation results, per-question paired statistics, and sampling settings are detailed in Appendix B.3 of the revised manuscript.

AuthorsOpenReview ↗

Kind Follow-up on Our Rebuttal Submission

Dear Reviewer zD62,

Thank you again for your thoughtful review.

We have provided detailed responses to your concerns and updated our manuscript with additional matched-backbone comparisons, mechanism-focused ablations, and per-question analyses.

As the discussion period is approaching its end, we would greatly appreciate it if you could take a look at our responses and let us know whether any concerns remain.

Thank you for your time and consideration of our responses in your final evaluation.

Best regards, The Authors

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer zD62,

As the discussion period is drawing to a close, we would like to briefly follow up on our rebuttal. We would greatly appreciate it if you could review our additional experiments and clarifications and let us know whether any concerns remain.

Thank you for your time and consideration.

Comments on the paper

1 comment

We sincerely thank all three reviewers for their thoughtful evaluations and constructive feedback.

We appreciate the reviewers' recognition of the following strengths:

  • [Reviewers zD62, Um7z] Practically relevant research problem.
  • [Reviewers zD62, iM8c, Um7z] Well-motivated design of Persistent Local Retrieval.
  • [Reviewer iM8c] Clear distinction between local retrieval adequacy and global answer sufficiency.
  • [Reviewers zD62, Um7z] Well-designed local–global information interface.
  • [Reviewers zD62, iM8c, Um7z] Convincing core ablation results.

The reviewers also raised questions regarding comparison fairness (Reviewers zD62, iM8c, Um7z), mechanism attribution (Reviewers zD62, iM8c, Um7z), information-boundary evaluation (Reviewers zD62, iM8c), experimental robustness and efficiency (Reviewers zD62, iM8c, Um7z), and prior-work distinctions and reproducibility (Reviewers zD62, iM8c, Um7z).

We addressed these comments through additional experiments, analyses, and revisions:

  • Matched-Backbone Comparison: Under the same backbone, tools, and budget, VideoPursuit outperforms an AVP-style retrieval agent by 6.7 percentage points (81.1% vs. 74.4%).
  • Mechanism Decomposition: Need-Locked improves accuracy from 68.7% to 73.5%, while complete Persistent Local Retrieval further reaches 81.1%, demonstrating the contributions of objective persistence and local retrieval control.
  • Information Boundary: Fixed-Trajectory Global Replay confirms a 5.1-point advantage for selective information delivery (81.1% vs. 76.0%) with acquired evidence held fixed.
  • Additional Evaluations: We added per-benchmark and paired per-question analyses, decoding robustness, computational cost measurements, navigation recovery, and general-purpose agent harness comparisons.
  • Technical and Prior-Work Clarifications: We expanded reproducibility details and comparisons with existing retrieval mechanisms.

Together, these additions further support the central contribution of VideoPursuit: transforming isolated Retrieval Actions into persistent, Need-Scoped Local Retrieval Processes.

We thank the reviewers and the Area Chair for considering our responses and the revised manuscript in their final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.