acceptodds
Withdrawn from ICLR 2027

VideoInspector: Agentic Multi-Video Reasoning with Compositional Segment Training

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

15 views

Abstract

Multimodal large language models have made substantial progress in video understanding, yet most are trained on single-video instruction data and struggle with multi-video reasoning. Such reasoning requires models to retrieve relevant videos, localize crucial moments, and aggregate evidence across different sources. However, these intermediate capabilities are rarely explicitly supervised by existing video training data, leading models to rely on coarse representations rather than grounded cross-video evidence. We introduce VideoInspector, an agentic framework for multi-video reasoning with tool-augmented segment inspection. Given a set of videos, VideoInspector performs coarse-to-fine reasoning by selecting relevant candidates, inspecting fine-grained temporal segments, and composing grounded evidence for final decisions. To train this model, we construct VideoInspector-Instruct, a large-scale segment-level compositional multi-video dataset. Starting from self-collected videos with dense temporal descriptions, we decompose videos into reusable semantic segments and recombine them into multi-video instances across four tasks: cross-video retrieval, cross-video temporal grounding, clip ordering, and video comparison. We further annotate tool-augmented reasoning trajectories and adopt a two-stage training pipeline: cold-start supervised fine-tuning teaches the model to imitate structured reasoning and tool use, while agentic reinforcement learning refines its policy for retrieval, inspection, and termination through exploration. Experiments on multi-video reasoning benchmarks show that VideoInspector improves over its non-tool variant on CrossVid, particularly in candidate analysis and temporal understanding. These results suggest that compositional segment supervision and tool-augmented reasoning provide a promising direction for building more grounded multi-video understanding systems.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Agentic Video Understanding, Video LLM, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_a2pGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Multi-video reasoning here is treated as a sequence of decisions about which video and temporal segment to inspect before answering. VideoInspector-Instruct supplies approximately 95K compositional examples covering retrieval, temporal grounding, clip reordering, and comparison, with synthetic tool-use and error-recovery trajectories. VideoInspector learns from these examples through supervised fine-tuning followed by agentic reinforcement learning. Evaluation on CrossVid, CVBench, and VEBench shows improvements over both backbone models and their trained variants without tools.

Critical strengths
  1. The gains extend across both backbones and all three benchmark summaries. CrossVid accuracy improves from 18.7 to 29.1 on the weaker backbone (Table 2), supporting the practical value of the supervision.

  2. The four training tasks cover complementary capabilities: retrieval, localization, ordering, and comparison. Removing any task reduces performance in Table 3, supporting the combined training design.

Critical weaknesses
  1. I am not convinced the gains establish better evidence acquisition and self-correction. The comparison without tools changes both supervision and access to segment observations; the stage ablation does not disentangle these effects. A comparison with fixed inspections under matched training and visual budgets would help. VEBench evaluates final localization, but intermediate evidence support and recovery remain untested quantitatively.

  2. Dataset validation remains too limited for a central contribution. Section 3.2 describes automatic filtering and manual spot checks, but reports neither the audit size nor measured annotation error rates. The number of unique source videos and benchmark overlap checks are also unspecified. A quantitative human audit and documented overlap checks would strengthen confidence in the supervision and reported transfer.

  3. Inference efficiency is not reported. Section 5.1 lists hardware and training settings, but gives no latency, average tool-call count, or inference cost. Repeated segment inspection could make the accuracy gains expensive. Without an accuracy–cost comparison against the baselines, I cannot assess whether the improvement justifies the additional computation.

Questions for authors
  1. What exactly does inspect_segment return during training and evaluation: sampled frames, generated descriptions, or both? What are the initial sampling settings and call limits, and how are visual budgets controlled across comparisons?

  2. Under matched training and visual budgets, how does learned inspection compare with fixed or uniformly selected inspections? Can you also report recovery success after an incorrect initial inspection, with and without recovery-trajectory supervision? These results could change my recommendation.

  3. How many unique videos underlie the dataset, what quantitative results were obtained from manual validation, and what checks excluded overlap with CrossVid, CVBench, and VEBench?

  4. On identical video–question inputs and comparable hardware, how do end-to-end latency, tool-call counts, and inference cost compare with the variant without tools and the strongest agentic baselines? Do the accuracy gains persist under the same time or compute budget?

Reviewer_GNbbOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes VideoInspector for multi-video reasoning. It introduces a 95K-example dataset covering retrieval, temporal grounding, clip reordering, and video comparison. The model uses a segment inspection tool and is trained with SFT and GRPO. Experiments evaluate two backbone models on several video benchmarks.

Critical strengths
  • The dataset covers several complementary multi-video tasks. This broadens the training setting beyond single-video question answering.

  • The paper evaluates two backbones and includes trained no-tool baselines. The training-stage and task ablations are also useful.

Critical weaknesses
  • Table 2 compares tool-enabled models with trained no-tool baselines. However, tool calls provide additional visual observations and computation. These budgets are not matched. The results therefore do not isolate the benefit of the proposed reasoning strategy.

  • Controlled mistakes are a central part of trajectory construction. Their contribution is not isolated against otherwise identical error-free trajectories. The paper also does not measure whether models actually recover from their mistakes.

  • The inspection tool is insufficiently specified. The returned observations, frame sampling, resolution, call limits, and stopping rules should be reported. These choices affect both accuracy and cost. The reward components and their combination also need clarification.

  • The data-quality audit is difficult to assess. Manual spot checks are mentioned, but the sample size and error rate are unclear. The paper should also document source diversity and overlap with evaluation videos.

  • Several numbers need correction. The easy and hard entries in Table 1 sum to 47K and 48K, rather than 44K and 51K. Table 4 reports 27.4 for tool-enabled SFT, while the text uses 27.0. This also changes the stated improvement.

Questions for authors
  • Does tool use still improve performance when visual observations and inference budgets are matched?

  • What is the isolated effect of training with recoverable mistakes?

  • What does each inspection call return? What are the average call count and inference cost?

  • How were annotation quality and train–evaluation overlap checked?

Reviewer_RUNCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces VIDEOINSPECTOR, a tool-augmented framework for multi-video reasoning, and VIDEOINSPECTOR-INSTRUCT, a 95K-sample compositional dataset. Videos are decomposed into timestamped semantic segments and recombined into four tasks: cross-video retrieval, temporal grounding, clip reordering, and video comparison. Training trajectories include inspect_segment tool calls and controlled, recoverable mistakes. The model is trained through tool-free/tool-augmented SFT followed by GRPO on about 8K uncertain examples. Experiments with Qwen2.5-VL-7B and Qwen3-VL-8B show improvements on CrossVid, CVBench, and VEBench.

Critical strengths
  1. The paper addresses an important and underexplored problem requiring retrieval, temporal localization, and evidence aggregation across videos.

  2. The four compositional tasks provide a reasonable decomposition of multi-video reasoning skills.

  3. Segment inspection enables coarse-to-fine evidence acquisition instead of uniformly processing all videos.

  4. Controlled error-recovery trajectories are an interesting way to teach replanning after plausible retrieval mistakes.

  5. Improvements are reasonably consistent across two Qwen-VL backbones and three benchmarks.

Critical weaknesses
  1. Dataset provenance and quality are insufficiently documented. The paper does not clearly specify video sources, licenses, release plans, or quantitative human audits. The pipeline relies on Gemini for segment annotation, GPT-4o for trajectories, and Qwen3-VL for filtering, but no detailed factual or temporal accuracy statistics are reported.

  2. Training-test contamination is unresolved. Since both training videos and benchmark videos may come from the internet, video-, frame-, subtitle-, and text-level deduplication against CrossVid, CVBench, and VEBench is essential but not reported.

  3. Tool comparisons are not compute controlled. The paper omits the maximum/average number of calls, returned frames, visual-token budget, latency, and FLOPs. Thus, gains over tool-free or differently configured agentic baselines may partly result from additional visual computation.

  4. Key components lack targeted ablations. The paper does not separately evaluate error-recovery trajectories, chain-of-thought supervision, uncertain-example selection, individual RL rewards, or direct tool quality such as evidence recall and inspected temporal IoU. Task-removal ablations also reduce total data size.

  5. Absolute performance remains limited. Tool use improves Overall by 1.9 points for Qwen2.5 and 3.0 points for Qwen3, but the Qwen3 model achieves only 0.031 on full VEBench OpSim versus 0.109 for Gemini-2.5-Pro. Claims of grounded multi-video reasoning should therefore be moderated.

  6. More visual result comparison or demos should be provided in Appendix. But It looks the authors do not upload any appendix.

Questions for authors
  1. What video sources and licenses are used, and exactly what data will be released?

  2. How many examples were manually audited, and what are the factual, temporal, trajectory, and answer accuracies?

  3. What deduplication was performed against the three benchmarks, and what are the results after removing possible overlaps?

  4. Do the gains remain under matched tool-call, visual-token, latency, or FLOP budgets?

  5. What are the independent contributions of error-recovery trajectories, SFT reasoning traces, uncertain-sample selection, and each RL reward?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.