VideoPursuit: From Retrieval Actions to Local Retrieval Processes for Agentic Long-Video Understanding

Agentic long-video understanding increasingly relies on adaptive video retrieval to access sparse, task-relevant observations during reasoning. However, many existing methods follow an action-level handoff pattern, returning control to global reasoning after each retrieval action. A useful retrieval action may still only narrow a temporal region, reveal a new cue, or indicate what should be inspected next, so completing the action does not necessarily mean that the current retrieval is ready for handoff. We introduce VideoPursuit, a framework that treats retrieval for each local objective as a dedicated process rather than a sequence of retrieval actions individually managed by global reasoning. When global reasoning over the original question identifies an information gap, it delegates a local retrieval objective to a dedicated retrieval agent. Within a bounded local execution scope, the agent may perform multiple retrieval actions around the same objective, using explicit local audit after each action to determine whether further retrieval is needed or the current result is ready for handoff. Intermediate retrieval states remain local, while acquired observations and their source information are selectively returned to global reasoning. Across four long-video understanding benchmarks, VideoPursuit achieves an average accuracy of 82.0%, improving by 3.4 percentage points over the strongest prior agentic method. Ablations further show that persistent local retrieval substantially outperforms action-level return, while selective information propagation across the local-global boundary provides additional gains.