acceptodds
Under review as a conference paper at ICLR 2027

Retool-Video: Recursive Tool-Using Video Agents with Meta-Augmented Tool Grounding

Abstract

Tool-augmented video agents typically require the ability to locate evidence, inspect multiple modalities, and integrate observations across long temporal horizons. However, existing approaches face two key challenges: at the *tool-space* level, video agents typically expose only a small set of coarse retrieval and inspection tools, limiting fine-grained evidence processing; at the *action-space* level, hierarchical video agents often leave underspecified how abstract video-reasoning intents are grounded into executable tool calls. To address these limitations, we introduce the MetaAug-Video Tool Library (MVTL), comprising 26 base tools for multimodal evidence acquisition and 108 meta tools for fine-grained operations over intermediate results. Building on MVTL, we propose **ReTool-Video**, a recursive tool-using framework in which a planner issues either executable tool calls or abstract intents, a resolver repairs, substitutes, or recursively decomposes abstract intents into validated tool calls, and the planner is optimized with reinforcement learning. Experimental results show that **ReTool-Video** achieves 72.9, 81.5, and 76.6 accuracy on MVBench, MLVU, and Video-MME, respectively. Under the same Qwen3.5-9B backbone and tool library without planner reinforcement learning, Planner–Resolver grounding improves MLVU accuracy by 13.1 percentage points over planner-only tool selection, demonstrating the benefit of separating high-level intent expression from low-level tool execution. The code and tools are available at [the anonymous repository](https://anonymous.4open.science/r/ReTool-Video).

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Understanding, Multimodal Agents, Tool Use, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_xnyzOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ReTool-Video, a video agent with a Planner and an LLM-based Resolver. The Planner decides what evidence is needed and when to answer. The Resolver turns these requests into actual tool calls by fixing arguments, choosing suitable tools, or breaking a request into smaller steps. The authors also build a library of 134 tools and train the Planner with GRPO. The system is evaluated on three video-understanding benchmarks.

Critical strengths
  1. The Planner can ask for the evidence it needs without getting every tool name and parameter right. The Resolver handles these details and returns the results for further reasoning.

  2. Without RL, adding it raises MLVU accuracy from 61.5% to 74.6%. The paper also tests another backbone, providing some evidence that the gains are not limited to Qwen models.

  3. Operations such as filtering and merging time intervals are useful for video questions. Removing meta tools lowers accuracy on all three benchmarks.

Critical weaknesses
  1. The Planner acts as the main agent controller, while the Resolver translates requests into tool calls. Argument repair, tool substitution, and recursive task decomposition are useful, but they build on familiar ideas from hierarchical agents.

  2. Freezing the Resolver provides a stable interface, as the authors explain, but a sensible Planner request can still be poorly translated or unnecessarily decomposed. The Planner may learn to phrase requests around these weaknesses, while the Resolver itself receives no training signal to improve. Since tool grounding is central to the method, the paper would benefit from explaining when this separation helps and when it becomes a bottleneck.

  3. The Resolver checks whether a tool call can run, but this does not ensure that it fulfils the Planner’s request: Appendix C.10 gives an example where frame-retrieval requests were repeatedly mapped to OCR without obtaining new evidence.

Questions for authors

On MLVU, w/o Recursion, in Table 3 scores 63.2%, while One-step grounding in Table 4 scores 72.8%. It is unclear how these settings differ, and the authors acknowledge that inference compute is not strictly matched. Figure 6 also assigns 58% of calls to visual/video tools, whereas Video_Clip_QA and Inspect_Frame alone total 81.55% in Table 8. Different settings or counting rules may explain these gaps, but they need to be stated so readers can interpret the results.

1 comment

Thanks for your response. My concerns have been addressed. Now I've increased my rating to 3.

Reviewer_FKEhOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes ReTool-Video, a recursive tool-using agent framework for complex video understanding tasks, aiming to address the limitations of existing video agents in terms of restricted tool diversity and the difficulty of mapping high-level reasoning intents to executable tool calls. The paper first introduces the MetaAug-Video Tool Library (MVTL), comprising 26 base tools and 108 meta tools to support multimodal evidence acquisition and fine-grained processing. Building on this library, the authors propose a Planner–Resolver collaborative mechanism, in which the Planner handles high-level task planning, while the Resolver translates abstract reasoning intents into executable tool chains through argument repair, tool substitution, and recursive decomposition. The Planner is further optimized through reinforcement learning. Experimental results demonstrate that ReTool-Video achieves accuracies of 72.9%, 81.5%, and 76.6% on MVBench, MLVU, and Video-MME, respectively, validating the effectiveness of recursive tool use and the separation of high-level planning from low-level tool execution in complex video understanding tasks.

Critical strengths
  1. Innovative and Well-Motivated Planner–Resolver Architecture The paper proposes a Planner–Resolver collaborative mechanism that decouples high-level video reasoning intents from low-level tool execution. The Planner is responsible for global task planning, while the Resolver translates abstract intents into executable tool chains through argument repair, tool substitution, and recursive decomposition. This design effectively addresses the difficulty of directly mapping abstract reasoning intents to specific tools in complex video reasoning tasks, improving the flexibility and reliability of tool execution.
  2. A Comprehensive and Highly Composable Tool Library The paper introduces the MetaAug-Video Tool Library (MVTL), comprising 26 base tools and 108 meta tools that support multimodal evidence acquisition, temporal merging, filtering, counting, reranking, and information aggregation. Compared with the limited set of coarse-grained tools commonly used by traditional video agents, MVTL provides a richer set of fine-grained operations, enabling agents to flexibly compose different tools to accomplish complex video understanding tasks. This design offers good extensibility and practical value.
  3. Comprehensive Experimental Results Demonstrating the Effectiveness of the Core Mechanisms The paper evaluates ReTool-Video on three video understanding benchmarks, including MVBench, MLVU, and Video-MME, achieving strong performance. In particular, under the same backbone model and tool environment, the Planner–Resolver mechanism improves MLVU accuracy from 61.5% to 74.6%, which further increases to 81.5% after reinforcement learning optimization. These results demonstrate that both recursive tool use and reinforcement learning contribute to improving video understanding performance, providing convincing empirical evidence for the effectiveness of the proposed framework.
Critical weaknesses
  1. Further Analysis of the Computational Overhead of the Framework The proposed Planner–Resolver mechanism involves recursive tool calls and multiple execution steps, which may introduce additional computational costs. The authors are encouraged to provide statistics on the number of tool calls, average inference time, and token consumption to enable a more comprehensive evaluation of the framework's efficiency and practicality in real-world applications.
  2. Further Analysis of Performance Across Different Video Question-Answering Tasks The paper achieves promising results on three benchmarks, namely MVBench, MLVU, and Video-MME. However, the authors are encouraged to further analyze performance differences across various question types, such as temporal localization, action counting, state tracking, and cross-modal reasoning. Such an analysis would help provide a clearer understanding of the strengths and limitations of ReTool-Video across different video reasoning tasks.
Questions for authors

The paper demonstrates significant performance improvements through the proposed Planner–Resolver mechanism. However, the Resolver introduces additional reasoning steps through recursive decomposition, tool substitution, and argument repair. If the planner-only baseline is allocated the same reasoning budget and number of tool calls, does the Planner–Resolver mechanism still maintain a significant advantage? The authors are encouraged to clarify whether the observed improvements primarily stem from the proposed hierarchical tool-grounding mechanism or from the additional inference computation.

Reviewer_98BhOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces MetaAug-Video Tool Library (MVTL) with 26 base tools and 108 meta tools for multi-modal evidence acquisition and fine-grained intermediate result processing. Building on MVTL, the authors propose ReTool-Video, an agentic framework for long-video understanding, which consists of a planner and a resolver. The planner issues either executable tool calls or abstract intents, and the resolver repairs, substitutes or recursively decomposes abstract intents into validated tool calls. The planner is further trained with reinforcement learning to improve the performance. Experimental results on MVBench, MLVU and Video-MME show that ReTool-Video outperforms the baseline methods and the proposed Planner-Resolver grounding framework effectively improves the performance by a significant margin.

Critical strengths
  1. The paper is well-written and easy to follow in general.
  2. The paper addresses the limitation of limited tools in tool-augmented video understanding, and proposes the Planner-Resolver framework which separates high-level planning from tool calling. This can reduce the burden on the Planner, especially there are a large number of tools with different interfaces.
  3. Experimental results on MVBench, MLVU and Video-MME demonstrates competitive performance compared to existing agentic video understanding methods. Ablation studies show the effectiveness on different components of the proposed methods.
Critical weaknesses
  1. The contribution of the large tool library is unclear. The proposed MVTL has 13 tools including 26 base tools and 108 meta tools. The five most frequently used tools account for 94.62% of all tool calls. It is unclear whether this large number of tools are necessary. A smaller tool library with carefully selected tools might achieve similar performance, and it might be easier for the model to use the tools correctly.
  2. The large tool pool and recursive grounding strategy might incur substantial computational cost compared to the baselines.
  3. The paper only trains the planner, while it is unclear that whether training resolver will bring additional performance gain since the introduction of the resolver seems to be one of the major contribution of the paper.
Questions for authors
  1. How does performance change when reducing the tool library to the most frequently used tools?
  2. Will training the resolver further improve the performance?
  3. Can the authors provide a computation cost comparison of ReTool-Video with the baselines?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.