- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies instance-level video question answering, where the target object or person is marked by a visual prompt (box, arrow, etc.) on a frame. The authors provide two tools: view_visual_prompt, which shows the marked frame, and crop_video, which returns a shorter clip for a closer look. The model learns to call these tools over multiple turns. The authors build an automatic four-stage pipeline on top of LLaVA-Video-178K to get 41k training samples, and collect 34.2k tool-use trajectories from Qwen3-VL-235B by rejection sampling for SFT, and run GRPO on 4.1k samples with accuracy, format, and tool-count rewards. On V2P-Bench, the 8B model improves from 60.8 to 74.5 and scores higher than GPT-4o and Gemini-2.5-Pro. It also gains a few points on Video-MME, LongVideoBench, and LongVT.
Critical strengths
-
The data pipeline is useful. This kind of data is hard to label by hand, and the uniqueness check handles the main difficulty (several similar objects in one scene). Releasing it would help the community.
-
Methodology effectiveness. A single-turn model trained on the same data reaches 67.3, below multi-turn SFT alone (70.4). So part of the gain does come from tool use, not just in-domain fine-tuning.
Critical weaknesses
-
The methodological contribution is thin, with standart tool set design and common train recipe. The main new part is the data pipeline, which does not support the "novel paradigm" framing.
-
The closed-source comparison is unfair and outdated. GPT-4o and Gemini-2.5-Pro are tested zero-shot, while VideoSeeker is trained on in-domain data. Both models are also no longer state of the art; Gemini-3.1-Pro is used in the authors' own pipeline but not evaluated. The claim in the abstract is not well supported.
-
The synthetic data is not validated. Errors can carry over across the automatic stages (e.g. a wrong tag leads to a mask on the wrong object), but only retention rates are reported. Rejection sampling cannot catch wrong labels. Since data is the main contribution, more in-depth analysis is needed.
Questions for authors
-
On the general benchmarks in Table 2 (Video-MME, LongVideoBench, LongVT), there is no visual prompt. What is given as F_vp in this case, and what does view_visual_prompt return if the model calls it? Does the model still call it, and how often?
-
Could you evaluate more recent closed models, such as claude opus 5 series, gpt-5/6 series, or Gemini-3.x-flash, especially those with agentic capabilities (rather than GPT-4o) on V2P-Bench?
5 comments
Responses to Official Review by Reviewer qYdC [2/5]
3. Motivation and practical value for real user needs
3.1 Visual prompts improve human-model interaction efficiency
As shown by the three examples in Figure 1 of the original submission, visual prompts address cumbersome and ambiguous textual reference in complex video scenes. Providing a visual prompt is not an additional burden: it directly expresses the target of interest, improves user experience and model understanding, and increases human-model interaction efficiency.
In long videos and complex multi-object scenes, users often need elaborate textual references. For example, in a Nobel Prize ceremony video, a user might ask, "In the first half of the video, what Nobel Prize did the N-th person from the left in the first row on stage receive, and in what order did they receive it?" This requires specifying time, position, and ordering, yet can remain ambiguous due to viewpoint changes, movement, or an unclear time range. A visual prompt lets the user directly mark the target in a reference frame and then ask the question.
For the model, text-only interaction first requires decoding the complex referring expression and grounding it across language and visual context before answering. Current base models remain weak at spatiotemporal grounding in these scenes and are prone to target-association errors. A visual prompt directly anchors the target, reducing ambiguity on both the user-expression and model-understanding sides.
We supplement this explanation with a text-versus-visual prompt comparison and a user experience study to quantify effects on model understanding and interaction. In the revision, we have added the settings, complete results, and analysis to Appendix C.5 (pp. 21–22, Table 15, lines 1113–1167).
(1) Visual prompts are more model-friendly. We rewrite the visual-prompt references in each QA pair as natural-language descriptions of the target's appearance and compare the same models under both prompt forms:
| Model | Text prompt | Visual prompt |
|---|---|---|
| GPT-4o | 53.0 | 65.4 |
| Gemini-2.5-Pro | 54.7 | 69.8 |
| LLaVA-Video-7B | 42.4 | 54.8 |
| Qwen2.5-VL-7B | 43.1 | 52.4 |
| Mimo-VL-7B | 46.7 | 55.6 |
Replacing visual reference with textual reference substantially decreases accuracy for every evaluated model, with a 15.1-percentage-point drop for Gemini-2.5-Pro. Text requires the model to decode the target solely from language, making user intent harder to interpret. Visual prompts instead specify the target directly in the frame, reducing the conversion burden from intent to text and then from text to a visual object.
(2) Visual prompts are more user-friendly. We supplement the evaluation with a user experience study:
| Metric | Text prompt | Visual prompt |
|---|---|---|
| Accuracy | 62.0 | 75.5 |
| Completion time (s) | 27.4 | 19.7 |
| User satisfaction | 5.8 | 8.2 |
With visual prompts, users finish tasks faster: completion time falls from 27.4 to 19.7 seconds, saving 7.7 seconds on average. Better model responses also improve satisfaction, which rises from 5.8 to 8.2, a 2.4-point increase. These results demonstrate clear advantages in both overall satisfaction and interaction efficiency.
Users can also choose text-only or visual prompts according to scene complexity. Visual prompts offer a more intuitive and efficient option for complex instance references.
Responses to Official Review by Reviewer qYdC [3/5]
3.2 The original submission discusses practical applications, and this interaction form is already deployed
As shown in Appendix A.2 (Applications of Instance-Level Video Understanding) of the original submission, we already discuss practical video interaction settings:
- Mobile-device interaction. Users watching videos on phones or computers can create visual prompts with a finger or mouse. The original submission cites systems such as GPT-4o and Video Refer to illustrate more intuitive interaction with video content and an improved user experience.
- Smart wearables. Devices such as Apple Vision Pro can support visual prompts through natural gestures to specify an object of interest, providing a natural entry point for interaction with immersive video content.
- Local or cloud model integration. The front end captures a reference frame and the user's annotation and passes them to a local or cloud vision-language model, enabling interaction about specific objects and turning passive viewing into active questioning and exploration.
This interaction form is already deployed in a real product. ByteDance's Doubao app lets users capture a video image of interest, mark a target through a doodle UI prompt, and ask questions about the visually specified instance. This "capture an image—mark a target—ask an instance-specific question" workflow directly reflects the real demand for specifying an object of interest through visual reference.
Our task thus directly serves real human-model interaction needs: the user indicates which instance they care about, and VideoSeeker uses native agentic multi-turn tool reasoning to retrieve evidence and answer about that instance. In the revision, we have added the interaction workflow of the Doubao case to Appendix A.2 (pp. 16–17, lines 860–865), further explaining the practical use of this interaction form.
In the revision, we have further clarified the meaning of the "new paradigm" and the first-of-its-kind combination in Section 2 (p. 4, lines 171–177), Section 3.1 (p. 4, lines 210–215), and Appendix A.3 (p. 17, lines 890–895). The heterogeneous distillation analysis providing general insights appears in Section 4.4 (p. 9, Figure 4, lines 472–485), and the interaction workflow of the Doubao case has been added to Appendix A.2 (pp. 16–17, lines 860–865).
Responses to Official Review by Reviewer qYdC [1/5]
We sincerely appreciate your insightful feedback and thoughtful suggestions. Below, we respond to each concern in detail and provide additional analysis and clarification.
Thank you for the feedback. We acknowledge that basic tool operations, SFT, and reinforcement learning are standard components. Here, the "new paradigm" refers to combining single-frame visual-prompt-driven instance-level video understanding with native agentic multi-turn active perception. To our knowledge, VideoSeeker is the first work to combine single-frame visual-prompt-driven instance-level video understanding with native agentic multi-turn tool reasoning. Its contributions lie in the combination of task input and reasoning interaction and in the systematic study, general insights, and practical value of this new topic.
1. The new paradigm combines visual reference with multi-turn active perception
As shown in Task Formulation And Environmental Interaction and Appendix A.3 (Task Novelty) of the original submission, the user specifies the target in one reference frame, and the model must understand that instance over time. Specifically, it revisits the target reference frame through
view_visual_prompt, obtains relevant temporal segments throughcrop_video, and interleaves visual observation and reasoning for the same question.We implement this paradigm with a large-scale automatic pipeline for QA filtering, target verification, mask generation, and visual-prompt rendering, followed by multi-turn tool-interaction trajectory generation. SFT and agentic RL internalize target association, reference-frame revisiting, and temporal evidence acquisition to support fine-grained visual-prompt-driven video QA.
The core of this paradigm is that users specify their object of interest visually, while the model actively selects subsequent observations around that instance and incorporates newly acquired visual evidence into reasoning about the same question. The data pipeline supports training, while task input, target association, and multi-turn visual interaction jointly form the methodological contribution.
2. Heterogeneous distillation across different teacher models
As shown in the Distillation Paradox analysis in the original submission, distilling a stronger Gemini teacher into a small Qwen model is less effective than distilling a large Qwen teacher from the same family.
Qwen3-VL-235B-A22B-Thinking and Gemini-3.1-Pro score 78.4% and 83.8% on the rejection-sampling dataset, respectively, favoring Gemini. However, using their trajectories for SFT of Qwen3-VL-8B produces students scoring 70.4% and 64.7% on V2P-Bench. A stronger teacher therefore does not yield a stronger student, contrary to the intuition that greater teacher capability necessarily leads to greater distillation gains.
Our analysis attributes this to differences in linguistic, reasoning, and tool-calling patterns across models. Same-family Qwen teachers and students have closer generation patterns, making useful trajectory information easier to absorb. With a heterogeneous teacher, a small student must learn the task while also adapting to different expression and reasoning patterns, increasing the learning burden and limiting capability transfer. This provides a practical insight for the community.