- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes the Visual Autonomous Investigation (VAI) task, in which an agent identifies the contextual meaning of an image and explores external information to provide explanations. Also, the authors propose Vision-AutonoSearch (VAS), a framework to address this task.
Critical strengths
- The authors propose a task called Visual Autonomous Investigation (VAI), where an agent independently selects what to investigate and uses external information (when given an image and a general description).
- The authors propose the Evidence-Constrained Comparative Reward (ECCR), which incorporates preference only when a core argument is substantiated by visual information or retrieved evidence, in addition to scores based on fixed evaluation criteria.
- The authors introduce Vision-AutonoSearch-9B that has been trained on VAI task. This model outperforms stronger models on the VAI task.
Critical weaknesses
- In my opinion, the task's novelty hasn't been fully explained. This paper defines VAI as a task where the agent determines its own investigation direction without being given a specific question to answer. However, if we had asked the search agent for a general request like "Explain this image" (like Figure 1), it's possible that we could have had similar autonomous investigations and explanations in the past. The paper compares existing agents, but doesn't directly compare their performance under the same general requests and search conditions, leaving unclear whether VAI demands new capabilities or simply applies new prompts and evaluation settings to existing search agents. To compare the VAS with existing agents, a controlled experiment is needed, using the same inputs, tools, and budget.
- I believe a more detailed explanation of the judge model is required. The authors verify alignment with humans in-house for only 20 responses. This is not sufficient to fully verify the judge's reliability.
- The authors used GPT-6 for dataset building, but why are the experiments conducted with GPT-5?
- Overall, I believe that existing agents have already been handling this task, even though it wasn't explicitly defined before. I think that's why the difference in score between the baseline and the Vision-AutonoSearch-9B isn't that significant. Overall, I believe this paper needs to be revised in terms of motivation, readability, and detail.
Questions for authors
- Why was Figure 1 placed on page 1? I think Section 4.2 above looks more better for readability.
- Which GPT-6 model was used for this work? (line 201) Also, in line 715, the author mentioned they used GPT-5.6-Sol, not GPT-6. Also, in the experiments, the author used GPT-5 (same, which GPT-5?). I was so confused about this.
- What happens if user ask like "Explain this image" the (strong) agent to describe this image in Figure 1? I think we can get similar response.
8 comments
Response to Reviewer PLey (Part 2/2)
W3 / Q2. Which GPT models did we use?
| Previous label | Exact model | Role |
|---|---|---|
| GPT-6 | GPT-6.1-Sol | Drafts reference explanations and rubrics for VAI-RL-2K. |
| GPT-5.6-Sol | GPT-5.6-Sol | Reviews VAIBench annotations alongside DeepSeek-V4-Flash. |
| GPT-5 | GPT-5.6-Sol | Serves as a Table 1 baseline, now evaluated with and without tools. |
We used GPT-6.1-Sol to help produce high-quality reference explanations and rubric drafts, and selected GPT-5.6-Sol as a frontier-model baseline. In revised Table 1, we also evaluate GPT-6.1-Sol with the same tools as VAS.
We have standardized the model names throughout the paper and clarified their roles in Appendix D.5. We also corrected the model name in Section 3.1 and Appendix A.1 to reflect that the SFT demonstrations were generated using GLM-5.3-Flash.
Q1. Figure placement and readability
Thank you for this constructive layout suggestion. We have moved the results overview to Figure 3 in Section 4.2 beside Table 1 to improve the flow of the experimental section. We use the new Figure 1 to introduce the task through The Threatened Swan, and have shortened the opening explanation to make the motivation easier to follow.
We sincerely appreciate your feedback. We hope the new controlled experiments, case study, and clearer presentation address your concerns, and we would be grateful if you considered updating your score.
Thanks to the authors for their response. However, the authors mention Appendix D.5, Appendix E, ..., in their response, but there are no corresponding Appendix sections. I think the authors need to check it again. Please let me know after resolving this issue.
Response to Reviewer PLey (Part 1/2)
Thank you for your detailed comments. We have clarified what VAI asks an agent to do, evaluated existing search agents under the same general request, expanded judge validation, and revised the figures and model descriptions.
W1 / W4 / Q3. Does VAI require more than a general image prompt?
We agree that a sufficiently strong general-purpose model can already investigate an image when asked to explain it. For example, GPT-5.6-Sol with our tools investigates the inscriptions in The Threatened Swan and explains their later political meaning using three sources. We now include this agentic frontier baseline in Table 1.
We emphasize that VAI asks the model to decide what to investigate and when the investigation is complete. In conventional search-augmented VQA, the user's question specifies the investigation target and provides a basis for judging whether the answer is sufficient. In VAI, the model must identify visual clues worth pursuing, choose an investigation direction, and then judge whether the available evidence supports a complete explanation.
We evaluated the specialized agents using the same request, “Help me understand this image,” with the same tools and budget. The following comparison on The Threatened Swan illustrates the different behaviors.
These trajectories show how the agents differ after identifying the painting, with VAS selecting the inscription as a further clue to explain. We provide the complete trajectories in Appendix G.1.
We also tested the same image with a specific question about the word “Holland” on the egg. The specialized agents then retrieved the relevant information, answered, and stopped. The contrast shows why choosing the investigation target matters. When the question supplies that target, the same search tools can find the answer.
We observe a similar pattern in our audit of 100 VAIBench images. Among the specialized-agent queries, 95% reformulate the main subject, compared with 21% for VAS-9B. Specialized agents investigate 1.1 distinct topics per image on average, compared with 2.8 for VAS. Their episodes reach the harness's finalization notice 82% of the time, compared with 9% for VAS. When given a specific question, the specialized agents' rate falls to 11%. Appendix F.2 reports the coding criteria and counts.
In the aggregate evaluation, the three specialized open baselines score 37.1 to 45.9 on VAI average. VAS-9B reaches 66.2 and outperforms them on all four VAI benchmarks. Within our training ablation, VAIBench rises from 67.1 with SFT to 69.9 with GRPO and 73.6 with ECCR. The trajectory analysis shows behavioral differences in choosing investigation targets and deciding when to stop, while the training ablation supports improvements in overall explanation quality. Together, these findings support dedicated training and evaluation for autonomous visual investigation.
W2. How reliable is the judge?
We expanded human validation from 20 responses to 200 responses covering all 100 VAIBench images, with two raters who did not know the models' identities or automatic scores. We also re-scored the responses with Claude Opus 5.5, an independent judge.
The human raters and both judges use the same criterion-level scoring rubric, assigning 0, 0.5, or 1 based on the report content and the evidence actually acquired by the agent.
Both the independent judge and human consensus confirm ECCR's improvement. Claude measures a 3.1-point VAIBench gain and human consensus a 3.0-point gain, with a 3.0-percentage-point UCR reduction in both evaluations. We include the full agreement protocol and results in Appendix E.