acceptodds
Under review as a conference paper at ICLR 2027

VideoSeeker: Incentivizing Instance-level Video Understanding via Native Agentic Tool Invocation

Abstract

Large Vision-Language Models (LVLMs) have shown significant progress in video understanding, yet they face substantial challenges in tasks requiring precise spatiotemporal localization at the instance level. Existing methods primarily rely on text prompts for human-model interaction, but these prompts struggle to provide precise spatial and temporal references, resulting in poor user experience. Furthermore, current approaches typically decouple visual perception from language reasoning, centering reasoning around language rather than visual content, which limits the model's ability to proactively perceive fine-grained visual evidence. To address these challenges, we propose VideoSeeker, a novel paradigm for instance-level video understanding through visual prompts. VideoSeeker seamlessly integrates agentic reasoning with instance-level video understanding tasks, enabling the model to proactively perceive and retrieve relevant video segments on demand. We construct a four-stage fully automated data synthesis pipeline to efficiently generate large-scale, high-quality instance-level video data. We internalize tool-calling and proactive perception capabilities into the model via cold-start supervision and RL training, building a powerful video understanding model. Experiments demonstrate that our model achieves an average improvement of +13.7% over baselines on instance-level video understanding tasks, surpassing powerful closed-source models such as GPT-4o and Gemini-2.5-Pro, while also showing effective transferability on general video understanding benchmarks. The relevant datasets and code will be released publicly.

Primary area:
foundation or frontier models, including LLMs
Keywords:
VLMs, Agent, RL

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_qYdCOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies instance-level video question answering, where the target object or person is marked by a visual prompt (box, arrow, etc.) on a frame. The authors provide two tools: view_visual_prompt, which shows the marked frame, and crop_video, which returns a shorter clip for a closer look. The model learns to call these tools over multiple turns. The authors build an automatic four-stage pipeline on top of LLaVA-Video-178K to get 41k training samples, and collect 34.2k tool-use trajectories from Qwen3-VL-235B by rejection sampling for SFT, and run GRPO on 4.1k samples with accuracy, format, and tool-count rewards. On V2P-Bench, the 8B model improves from 60.8 to 74.5 and scores higher than GPT-4o and Gemini-2.5-Pro. It also gains a few points on Video-MME, LongVideoBench, and LongVT.

Critical strengths
  1. The data pipeline is useful. This kind of data is hard to label by hand, and the uniqueness check handles the main difficulty (several similar objects in one scene). Releasing it would help the community.

  2. Methodology effectiveness. A single-turn model trained on the same data reaches 67.3, below multi-turn SFT alone (70.4). So part of the gain does come from tool use, not just in-domain fine-tuning.

Critical weaknesses
  1. The methodological contribution is thin, with standart tool set design and common train recipe. The main new part is the data pipeline, which does not support the "novel paradigm" framing.

  2. The closed-source comparison is unfair and outdated. GPT-4o and Gemini-2.5-Pro are tested zero-shot, while VideoSeeker is trained on in-domain data. Both models are also no longer state of the art; Gemini-3.1-Pro is used in the authors' own pipeline but not evaluated. The claim in the abstract is not well supported.

  3. The synthetic data is not validated. Errors can carry over across the automatic stages (e.g. a wrong tag leads to a mask on the wrong object), but only retention rates are reported. Rejection sampling cannot catch wrong labels. Since data is the main contribution, more in-depth analysis is needed.

Questions for authors
  1. On the general benchmarks in Table 2 (Video-MME, LongVideoBench, LongVT), there is no visual prompt. What is given as F_vp in this case, and what does view_visual_prompt return if the model calls it? Does the model still call it, and how often?

  2. Could you evaluate more recent closed models, such as claude opus 5 series, gpt-5/6 series, or Gemini-3.x-flash, especially those with agentic capabilities (rather than GPT-4o) on V2P-Bench?

5 comments
AuthorsOpenReview ↗

Responses to Official Review by Reviewer qYdC [1/5]

We sincerely appreciate your insightful feedback and thoughtful suggestions. Below, we respond to each concern in detail and provide additional analysis and clarification.

W1: The methodological contribution is thin, with standart tool set design and common train recipe. The main new part is the data pipeline, which does not support the "novel paradigm" framing.

Thank you for the feedback. We acknowledge that basic tool operations, SFT, and reinforcement learning are standard components. Here, the "new paradigm" refers to combining single-frame visual-prompt-driven instance-level video understanding with native agentic multi-turn active perception. To our knowledge, VideoSeeker is the first work to combine single-frame visual-prompt-driven instance-level video understanding with native agentic multi-turn tool reasoning. Its contributions lie in the combination of task input and reasoning interaction and in the systematic study, general insights, and practical value of this new topic.

1. The new paradigm combines visual reference with multi-turn active perception

As shown in Task Formulation And Environmental Interaction and Appendix A.3 (Task Novelty) of the original submission, the user specifies the target in one reference frame, and the model must understand that instance over time. Specifically, it revisits the target reference frame through view_visual_prompt, obtains relevant temporal segments through crop_video, and interleaves visual observation and reasoning for the same question.

We implement this paradigm with a large-scale automatic pipeline for QA filtering, target verification, mask generation, and visual-prompt rendering, followed by multi-turn tool-interaction trajectory generation. SFT and agentic RL internalize target association, reference-frame revisiting, and temporal evidence acquisition to support fine-grained visual-prompt-driven video QA.

The core of this paradigm is that users specify their object of interest visually, while the model actively selects subsequent observations around that instance and incorporates newly acquired visual evidence into reasoning about the same question. The data pipeline supports training, while task input, target association, and multi-turn visual interaction jointly form the methodological contribution.

2. Heterogeneous distillation across different teacher models

As shown in the Distillation Paradox analysis in the original submission, distilling a stronger Gemini teacher into a small Qwen model is less effective than distilling a large Qwen teacher from the same family.

Qwen3-VL-235B-A22B-Thinking and Gemini-3.1-Pro score 78.4% and 83.8% on the rejection-sampling dataset, respectively, favoring Gemini. However, using their trajectories for SFT of Qwen3-VL-8B produces students scoring 70.4% and 64.7% on V2P-Bench. A stronger teacher therefore does not yield a stronger student, contrary to the intuition that greater teacher capability necessarily leads to greater distillation gains.

Our analysis attributes this to differences in linguistic, reasoning, and tool-calling patterns across models. Same-family Qwen teachers and students have closer generation patterns, making useful trajectory information easier to absorb. With a heterogeneous teacher, a small student must learn the task while also adapting to different expression and reasoning patterns, increasing the learning burden and limiting capability transfer. This provides a practical insight for the community.

AuthorsOpenReview ↗

Responses to Official Review by Reviewer qYdC [2/5]

3. Motivation and practical value for real user needs

3.1 Visual prompts improve human-model interaction efficiency

As shown by the three examples in Figure 1 of the original submission, visual prompts address cumbersome and ambiguous textual reference in complex video scenes. Providing a visual prompt is not an additional burden: it directly expresses the target of interest, improves user experience and model understanding, and increases human-model interaction efficiency.

In long videos and complex multi-object scenes, users often need elaborate textual references. For example, in a Nobel Prize ceremony video, a user might ask, "In the first half of the video, what Nobel Prize did the N-th person from the left in the first row on stage receive, and in what order did they receive it?" This requires specifying time, position, and ordering, yet can remain ambiguous due to viewpoint changes, movement, or an unclear time range. A visual prompt lets the user directly mark the target in a reference frame and then ask the question.

For the model, text-only interaction first requires decoding the complex referring expression and grounding it across language and visual context before answering. Current base models remain weak at spatiotemporal grounding in these scenes and are prone to target-association errors. A visual prompt directly anchors the target, reducing ambiguity on both the user-expression and model-understanding sides.

We supplement this explanation with a text-versus-visual prompt comparison and a user experience study to quantify effects on model understanding and interaction. In the revision, we have added the settings, complete results, and analysis to Appendix C.5 (pp. 21–22, Table 15, lines 1113–1167).

(1) Visual prompts are more model-friendly. We rewrite the visual-prompt references in each QA pair as natural-language descriptions of the target's appearance and compare the same models under both prompt forms:

ModelText promptVisual prompt
GPT-4o53.065.4
Gemini-2.5-Pro54.769.8
LLaVA-Video-7B42.454.8
Qwen2.5-VL-7B43.152.4
Mimo-VL-7B46.755.6

Replacing visual reference with textual reference substantially decreases accuracy for every evaluated model, with a 15.1-percentage-point drop for Gemini-2.5-Pro. Text requires the model to decode the target solely from language, making user intent harder to interpret. Visual prompts instead specify the target directly in the frame, reducing the conversion burden from intent to text and then from text to a visual object.

(2) Visual prompts are more user-friendly. We supplement the evaluation with a user experience study:

MetricText promptVisual prompt
Accuracy62.075.5
Completion time (s)27.419.7
User satisfaction5.88.2

With visual prompts, users finish tasks faster: completion time falls from 27.4 to 19.7 seconds, saving 7.7 seconds on average. Better model responses also improve satisfaction, which rises from 5.8 to 8.2, a 2.4-point increase. These results demonstrate clear advantages in both overall satisfaction and interaction efficiency.

Users can also choose text-only or visual prompts according to scene complexity. Visual prompts offer a more intuitive and efficient option for complex instance references.

AuthorsOpenReview ↗

Responses to Official Review by Reviewer qYdC [3/5]

3.2 The original submission discusses practical applications, and this interaction form is already deployed

As shown in Appendix A.2 (Applications of Instance-Level Video Understanding) of the original submission, we already discuss practical video interaction settings:

  • Mobile-device interaction. Users watching videos on phones or computers can create visual prompts with a finger or mouse. The original submission cites systems such as GPT-4o and Video Refer to illustrate more intuitive interaction with video content and an improved user experience.
  • Smart wearables. Devices such as Apple Vision Pro can support visual prompts through natural gestures to specify an object of interest, providing a natural entry point for interaction with immersive video content.
  • Local or cloud model integration. The front end captures a reference frame and the user's annotation and passes them to a local or cloud vision-language model, enabling interaction about specific objects and turning passive viewing into active questioning and exploration.

This interaction form is already deployed in a real product. ByteDance's Doubao app lets users capture a video image of interest, mark a target through a doodle UI prompt, and ask questions about the visually specified instance. This "capture an image—mark a target—ask an instance-specific question" workflow directly reflects the real demand for specifying an object of interest through visual reference.

Our task thus directly serves real human-model interaction needs: the user indicates which instance they care about, and VideoSeeker uses native agentic multi-turn tool reasoning to retrieve evidence and answer about that instance. In the revision, we have added the interaction workflow of the Doubao case to Appendix A.2 (pp. 16–17, lines 860–865), further explaining the practical use of this interaction form.

In the revision, we have further clarified the meaning of the "new paradigm" and the first-of-its-kind combination in Section 2 (p. 4, lines 171–177), Section 3.1 (p. 4, lines 210–215), and Appendix A.3 (p. 17, lines 890–895). The heterogeneous distillation analysis providing general insights appears in Section 4.4 (p. 9, Figure 4, lines 472–485), and the interaction workflow of the Doubao case has been added to Appendix A.2 (pp. 16–17, lines 860–865).

Reviewer_3PpnOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper builds a data engine by:

  • gpt, gemini, sam 3 etc to filter and mask to generate the vqa samples
  • qwen3vl 235b a22b thinking to generate the sft trajectories which will be further filtered down to rl data

The data contains tool calls

  • view_visual_prompt to retrieve the instance masked image
  • crop_video to locate a specific time range of the video

The data will be used to further posttrain a agentic video model on top of the qwen3vl 4b or 8b, named VideoSeeker.

The model then evaluated on the v2p bench, showing that the posttrain helps to boost the performance +11.4 and +13.7 on v2p bench for 4b and 8b model respectively

Critical strengths
  • A clear design of the data engine and how to use that data to train a agentic model. The results showed improvements that a tool call posttrained model can help v2p bench and several other general benchmarks like videomme, longvideobench etc.
Critical weaknesses
  • 1\ Limited novelty
    • New insights from the paper is limited. The technique is commonly used in academia and industry. Stronger / larger model combined to generate SFT data, reject sampling to generate RL data, then a baseline model will get performance improvement by training on the data. This paper didn't add new things.
  • 2\ Practically limited also
    • This work has one clear limitation that it needs the benchmark have the pre-drawn images or the user needs to give the visual prompts. On the spatial grounding part, the additional capability introduced by the work is basically just from the larger qwen teacher model regarding when to retrieve the already ready-to-use visual prompt. While many models nowadays already support grounding capability directly. The practical usage of this model also became a concern.
Questions for authors
  • when evaluating on other benchmarks (not v2p) where there is no pre-drawn masked images, hows the tool call view_visual_prompt will be leveraged. do we have online gemini + sam3 to generate the masked image or it is just ignored?
7 comments
AuthorsOpenReview ↗

Responses to Official Review by Reviewer 3Ppn [1/4]

We sincerely appreciate your insightful feedback and thoughtful suggestions. Below, we respond to each concern in detail and provide additional analysis and clarification.

W1: New insights from the paper is limited. The technique is commonly used in academia and industry. Stronger / larger model combined to generate SFT data, reject sampling to generate RL data, then a baseline model will get performance improvement by training on the data. This paper didn't add new things.

Thank you for the feedback. We acknowledge that teacher-generated data, SFT, and reinforcement learning are standard technical components. Our contributions nevertheless lie in systematically studying this new topic, its concrete implementation, general insights, and practical value, rather than simply choosing these components. To our knowledge, VideoSeeker is the first work to combine single-frame visual-prompt-driven instance-level video understanding with native agentic multi-turn tool reasoning.

1. The first visual-prompt-driven multi-turn instance-level active perception framework

As shown in Task Formulation And Environmental Interaction and Appendix A.3 (Task Novelty) of the original submission, the user specifies a target in just one reference frame, and the model must understand that instance over time. Specifically, view_visual_prompt revisits the target reference frame, while crop_video retrieves relevant temporal segments, interleaving visual observation and reasoning for the same question. We build a large-scale automatic data pipeline comprising QA filtering, target verification, mask generation, and visual-prompt rendering, and further generate multi-turn tool-interaction trajectories. SFT and agentic RL then internalize target association, reference-frame revisiting, and temporal evidence acquisition to support fine-grained visual-prompt-driven video QA.

2. Heterogeneous distillation across different teacher models

As shown in the Distillation Paradox analysis in the original submission, a stronger Gemini teacher produces a weaker small Qwen student than a large Qwen teacher from the same family. Qwen3-VL-235B-A22B-Thinking and Gemini-3.1-Pro score 78.4% and 83.8% on the rejection-sampling dataset, respectively, favoring Gemini. However, after their trajectories are used for SFT of Qwen3-VL-8B, the students score 70.4% and 64.7% on V2P-Bench. The stronger teacher therefore does not yield a stronger student, contrary to the intuition that higher teacher capability necessarily yields greater distillation gains.

Our analysis attributes this to differences in linguistic, reasoning, and tool-calling patterns across models. Same-family Qwen teachers and students have closer generation patterns, making useful trajectory information easier to absorb. With a heterogeneous teacher, a small student must learn the task while also adapting to different expression and reasoning patterns, increasing the learning burden and limiting capability transfer. This provides a practical insight for the community.

3. Practical value for real user needs

As shown in Appendix A.2 (Applications of Instance-Level Video Understanding) of the original submission, the task directly serves natural video human-model interaction. A user watching a video on a phone or computer can circle a person or object in one frame with a touchscreen or mouse and ask about its later behavior; wearable devices can similarly use natural gestures. For example, circling a person and asking "What did this person pick up later?" avoids first describing their appearance, clothing, and location or annotating every frame.

VideoSeeker leaves subsequent instance association and evidence acquisition to the model, allowing one visual reference to support questions about an instance over time. This input form is already deployed in a real product: the Doubao app lets users capture a video image, specify an instance through a doodle prompt in the interface, and ask questions about it. This directly illustrates the practical demand for visual-reference-based interaction.

In the revision, we have further clarified the first-of-its-kind combination in Section 2 (p. 4, lines 171–177) and Appendix A.3 (p. 17, lines 890–895). The heterogeneous distillation analysis providing general insights remains in Section 4.4 (p. 9, Figure 4, lines 472–485). We have also added the interaction workflow of the Doubao case in Appendix A.2 (pp. 16–17, lines 860–865).

AuthorsOpenReview ↗

Responses to Official Review by Reviewer 3Ppn [2/4]

W2: This work has one clear limitation that it needs the benchmark have the pre-drawn images or the user needs to give the visual prompts. On the spatial grounding part, the additional capability introduced by the work is basically just from the larger qwen teacher model regarding when to retrieve the already ready-to-use visual prompt. While many models nowadays already support grounding capability directly. The practical usage of this model also became a concern.

Thank you for the feedback. We clarify the interaction value of visual prompts, the capabilities actually learned by the model, and concrete application settings.

1. Visual prompts improve human-model interaction efficiency

As shown by the three examples in Figure 1 of the original submission, visual prompts address cumbersome and ambiguous textual reference in complex video scenes. Providing a visual prompt is not an additional burden: it directly expresses the target of interest, improves user experience and model understanding, and increases human-model interaction efficiency.

In long videos and complex multi-object scenes, users often need elaborate textual references. For example, in a Nobel Prize ceremony video, a user might ask, "In the first half of the video, what Nobel Prize did the N-th person from the left in the first row on stage receive, and in what order did they receive it?" This requires specifying time, position, and ordering, yet can remain ambiguous due to viewpoint changes, movement, or an unclear time range. A visual prompt lets the user directly mark the target in a reference frame and then ask the question.

For the model, text-only interaction first requires decoding the complex referring expression and grounding it across language and visual context before answering. Current base models remain weak at spatiotemporal grounding in these scenes and are prone to target-association errors. A visual prompt directly anchors the target, reducing ambiguity on both the user-expression and model-understanding sides.

We supplement this explanation with a text-versus-visual prompt comparison and a user experience study to quantify effects on model understanding and interaction. In the revision, we have added the settings, complete results, and analysis to Appendix C.5 (pp. 21–22, Table 15, lines 1113–1167).

(1) Visual prompts are more model-friendly. We rewrite the visual-prompt references in each QA pair as natural-language descriptions of the target's appearance and compare the same models under both prompt forms:

ModelText promptVisual prompt
GPT-4o53.065.4
Gemini-2.5-Pro54.769.8
LLaVA-Video-7B42.454.8
Qwen2.5-VL-7B43.152.4
Mimo-VL-7B46.755.6

Replacing visual reference with textual reference decreases accuracy for every evaluated model; Gemini-2.5-Pro drops by 15.1 percentage points. Text requires the model to identify the target from language alone, making the user's intent harder to interpret. Visual prompts directly specify the target in the frame, reducing the conversion burden from user intent to text and then from text to a visual object.

(2) Visual prompts are more user-friendly. We supplement the evaluation with a user experience study:

MetricText promptVisual prompt
Accuracy62.075.5
Completion time (s)27.419.7
User satisfaction5.88.2

With visual prompts, users finish tasks faster: completion time falls from 27.4 to 19.7 seconds, saving 7.7 seconds on average. Better model responses also improve satisfaction, which rises from 5.8 to 8.2, a 2.4-point increase. These results demonstrate clear advantages in both overall satisfaction and interaction efficiency.

Users can also choose text-only or visual prompts according to scene complexity. Visual prompts offer a more intuitive and efficient option for complex instance references.

AuthorsOpenReview ↗

Responses to Official Review by Reviewer 3Ppn [3/4]

2. VideoSeeker learns more than reference-frame retrieval

As shown in Task Formulation And Environmental Interaction of the original submission, the model uses view_visual_prompt to inspect the target reference frame and crop_video to obtain relevant temporal segments, then continues reasoning using returned visual evidence. The original tool ablation and direct temporal grounding evaluation quantify tool complementarity and localization ability, respectively.

Tool ablation. As shown in Table 3 (Tools Ablation) of the original submission:

Settingview_visual_promptcrop_videoV2P-Bench Accuracy
Qwen3-VL-8B (Base)✗✗60.8
VideoSeeker tool ablation✓✗69.4
VideoSeeker tool ablation✗✓63.7
Full VideoSeeker✓✓74.5

Combining both tools reaches 74.5, compared with 69.4 using the reference-frame tool alone, showing the additional benefit of temporal evidence acquisition beyond target reference.

Direct temporal grounding evaluation. As shown in Appendix C.2 (Temporal Grounding Evaluation) of the original submission, we also evaluate directly on three temporal grounding benchmarks:

ModelCharades-STA mIoUActivityNet mIoUQVHighlights mIoUAvg. mIoU
Qwen3-VL-4B Base37.133.838.936.6
+Single Turn SFT+RL (4B)32.429.234.231.9
VideoSeeker-4B41.236.942.540.2
Qwen3-VL-8B Base39.435.641.038.7
+Single Turn SFT+RL (8B)35.131.036.334.1
VideoSeeker-8B44.539.245.343.0

VideoSeeker outperforms both the corresponding base model and single-turn control across all three benchmarks, providing evidence of temporal grounding ability independently of QA accuracy.

The capabilities learned by this work therefore include not only when to retrieve the reference frame, but also target-specific instance association, active temporal cropping, and multi-turn reasoning with newly acquired visual evidence. In the revision, we have further clarified these capabilities and their evidence in Section 3.1 (p. 4, lines 210–215), Section 4.3 (p. 8, Table 3), and Appendix C.2 (p. 20, Table 13).

3. The original submission explains practical applications, and this interaction form is already deployed

As shown in Appendix A.2 (Applications of Instance-Level Video Understanding) of the original submission, we already discuss practical video interaction settings:

  • Mobile-device interaction. Users watching videos on phones or computers can create visual prompts with a finger or mouse. The original submission cites systems such as GPT-4o and Video Refer to illustrate more intuitive interaction with video content and an improved user experience.
  • Smart wearables. Devices such as Apple Vision Pro can support visual prompts through natural gestures to specify an object of interest, providing a natural entry point for interaction with immersive video content.
  • Local or cloud model integration. The front end captures a reference frame and the user's annotation and passes them to a local or cloud vision-language model, enabling interaction about specific objects and turning passive viewing into active questioning and exploration.

This interaction form is already deployed in a real product. ByteDance's Doubao app lets users capture a video image of interest, mark a target through a doodle UI prompt, and ask questions about the visually specified instance. This "capture an image—mark a target—ask an instance-specific question" workflow directly reflects the real demand for specifying an object of interest through visual reference.

Our task thus directly serves real human-model interaction needs: the user indicates which instance they care about, and VideoSeeker uses native agentic multi-turn tool reasoning to retrieve evidence and answer about that instance. In the revision, we have added the interaction workflow of the Doubao case to Appendix A.2 (pp. 16–17, lines 860–865), further explaining the practical use of this interaction form.

Reviewer_Qv88OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

VideoSeeker combines reference-image viewing and temporal video cropping with an automated visual-prompt QA synthesis pipeline. Through supervised fine-tuning and agentic reinforcement learning, it learns to interact with videos conditioned on specific instances and improves performance on instance-level QA and general video benchmarks.

Critical strengths
  1. The data pipeline is a solid engineering contribution. It offers a practical way to scale instance-conditioned supervision, producing approximately 41k visual-prompt QA examples and 34.2k multi-turn trajectories across eight prompt types. Consistent gains across model families indicate that the value of this data and training resource extends beyond a single backbone.
  2. Comprehensive experiments and ablations. The experiments also show that multi-turn visual inspection offers benefits beyond simply fine-tuning on the target task. The single-turn SFT+RL control scores 67.3, compared with 74.5 for the agentic model. This comparison provides meaningful evidence that revisiting visual evidence improves performance.
Critical weaknesses
  1. How does the proposed training method advance beyond applying an existing native video-agent recipe to visual-prompt data? INST-IT already studies visual-prompt-based instance instruction tuning, while LongVT combines native temporal cropping with SFT and agentic RL. Beyond the new data construction, what methodological contribution supports the claimed paradigm advance?
  2. The paper’s central claim has not been directly measured. It claims to achieve “precise localization,” yet it does not evaluate the temporal accuracy of the cropped video. I would like to know what quantitative metrics are used to assess VideoSeeker’s crop accuracy, for example through direct evaluation on video temporal grounding benchmarks.
Questions for authors
  1. What practical applications does instance-level video understanding enable? More fundamentally, how can the authors establish that this task addresses a real need rather than being designed for its own sake?
  2. Does the gain come from the training data or from multi-turn tool use? I would like to see the performance of single-turn and multi-turn training, so that the advantage of the multi-turn agentic paradigm can be better quantified.
  3. Do the single-turn controls receive the annotated reference frame directly? After ablating view_visual_prompt, what reference information remains available to the model?
  4. Are the synthesized training videos disjoint from V2P-Bench and the temporal-grounding evaluation sets? This is particularly important for the Charades and ActivityNet sources listed in Table 8.
  5. To what extent do the synthesized questions and model predictions depend on the visual referent? What happens when the prompt is removed?
9 comments
AuthorsOpenReview ↗

Responses to Official Review by Reviewer Qv88 [1/6]

We sincerely appreciate your insightful feedback and thoughtful suggestions. Below, we respond to each concern in detail and provide additional analysis and clarification.

W1: How does the proposed training method advance beyond applying an existing native video-agent recipe to visual-prompt data? INST-IT already studies visual-prompt-based instance instruction tuning, while LongVT combines native temporal cropping with SFT and agentic RL. Beyond the new data construction, what methodological contribution supports the claimed paradigm advance?

1. Specific differences from INST-IT and LongVT

Our differences from INST-IT concern the input form of visual reference, active acquisition of visual evidence through multi-turn tool interaction, and practical applicability. INST-IT provides instance references in all input video frames and generates a single-turn answer from the given visual input, without tool calls or active acquisition of new visual evidence during multi-turn reasoning. By contrast, to our knowledge, VideoSeeker is the first work to combine visual-prompt-driven instance-level video understanding with native agentic multi-turn tool reasoning. Starting from a target specified in a single reference frame, the model revisits the reference frame, crops relevant video segments, and continues reasoning using newly acquired visual evidence for the same question.

This difference also corresponds to different human-model interaction requirements. In practice, users typically indicate an object of interest in one frame rather than provide consistent instance IDs for every frame of an entire video, which is infeasible in the intended application setting. Thus, the cross-frame marked input used by INST-IT conflicts with the practical interaction setting in which a user marks the target only once. VideoSeeker directly addresses the latter setting, leaving the association between the local visual reference and video evidence to the model.

Our difference from LongVT is whether temporal evidence search is constrained by the target instance visually specified by the user. LongVT already uses native temporal cropping and multi-turn reasoning to search long videos for question-relevant evidence. VideoSeeker further studies how to search around a target when it is specified through a visual prompt rather than a sufficiently detailed textual description. In scenes with multiple similar instances, finding a semantically relevant event or time segment is insufficient: the model must also establish that the evidence belongs to the intended instance. VideoSeeker therefore uses the reference frame as a visual anchor of target identity, allowing the model to combine the target reference with temporal cues to decide when to revisit the reference frame, where to crop the video, and how to answer using the returned visual evidence.

Accordingly, our work integrates target association and on-demand temporal evidence acquisition into a single active perception process for instance-level understanding from a single-frame visual reference.

2. Contributions beyond data construction

First, the methodological contribution is an active perception paradigm centered on instance reference. We organize visual target association, reference-frame revisiting, and temporal segment inspection into a learnable interaction policy. This directly supports the interaction in which a user marks an object and asks about its behavior over time.

Second, this paradigm supports natural, low-effort video human-model interaction. As discussed in Appendix A.2, a user can specify a target through a phone touchscreen, a computer mouse, or a wearable-device gesture and directly ask, for example, "What did this person pick up later?" This enables questions about an instance of interest without frame-by-frame annotation or complex textual reference.

Finally, our work reveals heterogeneous distillation behavior across teacher models. We find that distilling a stronger Gemini teacher into a small Qwen model is less effective than distilling a large Qwen teacher from the same family. Our analysis attributes this to differences in linguistic and reasoning patterns that affect the student's absorption of teacher trajectories. This yields an insight beyond a particular model: teacher selection should consider not only the teacher's task performance, but also teacher-student pattern compatibility and actual transfer effectiveness.

We have clarified the differences between VideoSeeker, INST-IT, and LongVT and the first-of-its-kind combination in Section 2 and Appendix A.3 of the revision. Practical applications are discussed in Appendix A.2, and the heterogeneous distillation analysis appears in Section 4.4.

AuthorsOpenReview ↗

Responses to Official Review by Reviewer Qv88 [2/6]

W2: The paper’s central claim has not been directly measured. It claims to achieve “precise localization,” yet it does not evaluate the temporal accuracy of the cropped video. I would like to know what quantitative metrics are used to assess VideoSeeker’s crop accuracy, for example through direct evaluation on video temporal grounding benchmarks.

We agree that localization accuracy should be measured directly. As shown in Table 13 in Appendix C.2 (Temporal Grounding Evaluation) of the original submission, we already directly evaluate temporal grounding on three classic video temporal grounding (VTG) benchmarks—Charades-STA, ActivityNet Captions, and QVHighlights—following the evaluation protocol of recent MLLM-based VTG methods and reporting metrics at IoU thresholds of 0.3, 0.5, and 0.7, together with mIoU:

ModelCharades-STA @0.3@0.5@0.7mIoUActivityNet @0.3@0.5@0.7mIoUQVHighlights @0.3@0.5@0.7mIoUAvg. mIoU
Qwen3-VL-4B (Base)54.239.819.337.146.329.715.233.855.841.222.638.936.6
+ SFT+RL (Single Turn,4B)49.134.615.232.441.324.911.429.250.536.118.534.231.9
VideoSeeker-4B58.944.623.141.250.133.217.936.960.346.126.442.540.2
Qwen3-VL-8B (Base)57.142.521.839.448.932.116.835.658.444.024.941.038.7
+ SFT+RL (Single Turn,8B)52.337.617.735.143.727.312.631.053.438.820.236.334.1
VideoSeeker-8B62.748.326.944.553.436.520.339.263.949.729.245.343.0

As shown, even without any VTG-specific training, VideoSeeker consistently improves localization accuracy over its base model across all three benchmarks: average mIoU increases from 36.6 to 40.2 (+3.6) for 4B, and from 38.7 to 43.0 (+4.3) for 8B. These results directly and quantitatively validate temporal grounding ability instead of relying solely on downstream QA accuracy as indirect evidence.

The single-turn control is particularly informative: single-turn SFT+RL falls below the base model, with average mIoU decreasing from 36.6 to 31.9 (−4.7) for 4B and from 38.7 to 34.1 (−4.6) for 8B. This indicates that the localization improvement comes from multi-turn agentic reinforcement learning rather than training alone. This is consistent with the mechanism of crop_video: actively cropping candidate temporal windows and re-inspecting local segments at a higher effective frame rate helps improve localization accuracy. The complete results and analysis are already included in Appendix C.2 of the original submission; in the revision, we have further clarified the metrics and results in Section 4.3 and Appendix C.2 (Table 13, p. 20, lines 1052–1055) and strengthened the main-text reference to this evaluation.

AuthorsOpenReview ↗

Responses to Official Review by Reviewer Qv88 [3/6]

Q1: What practical applications does instance-level video understanding enable? More fundamentally, how can the authors establish that this task addresses a real need rather than being designed for its own sake?

As discussed in Appendix A.2 (Applications of Instance-Level Video Understanding) of the original submission, we already explain the practical value of this task in video human-model interaction. It addresses a natural user need: directly indicating an object of interest and asking questions about it, without first fully describing the target in words or annotating it frame by frame. We discuss two application settings:

  • Mobile-device interaction: When watching videos on a phone or computer, users can create a visual prompt with a finger or mouse, directly identify a person or object of interest, and ask questions about the video. This provides a more intuitive way to express the target and improves the interaction experience.
  • Smart wearables: Users can generate visual prompts through natural gestures to specify a target and interact with the device's vision-language model, providing a natural entry point for instance-level QA in immersive video content.

In these settings, the front end captures the reference frame, visual prompt, and question and passes the user's reference to a local or cloud model. VideoSeeker further performs target association and on-demand video evidence acquisition, enabling questions about the specified instance's behavior and changes over time.

This need is already reflected in a deployed product. ByteDance's Doubao app provides the corresponding interaction: after a user captures a video image of interest, the interface prompts the user to mark a target with a doodle and then ask questions about that visually specified instance. This "capture an image—mark the target—ask an instance-specific question" workflow directly reflects the practical need to express the object of interest through a visual prompt. Our task therefore arises from real interaction needs: the prompt establishes which instance the user means, while instance-level video understanding answers what happens to that instance in the video. The Doubao example demonstrates the reality of this interaction need, and VideoSeeker systematically studies how native agentic multi-turn tool reasoning can address it.

In the revision, we have added the Doubao product example to Appendix A.2 (pp. 16–17, lines 860–865), further illustrating the practical adoption of visual-prompt-driven instance QA.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.