acceptodds
Under review as a conference paper at ICLR 2027

Vision-AutonoSearch: Learning to Investigate Images

Abstract

Multimodal search agents answer knowledge-intensive visual questions through multi-turn search, but the question typically already specifies the search direction. Many images, such as memes, advertisements, artworks, and news photographs, instead derive meaning from the context they reference or imply, so explaining them requires deciding what to investigate. We formulate this task as Visual Autonomous Investigation (VAI): given an image and a general explanation request, the agent decides what to investigate, gathers evidence, and produces a grounded explanation. We present Vision-AutonoSearch (VAS), a framework that trains a single policy for VAI and search-augmented VQA through supervised fine-tuning and reinforcement learning. We build VAI-SFT-5K with investigation demonstrations and VAI-RL-2K with private rubrics and reference evidence. Rubrics score essential content but cannot distinguish explanations that meet the same criteria yet differ in accuracy, evidential support, or additional findings. We therefore propose Evidence-Constrained Comparative Reward (ECCR), which complements rubric scores with pairwise comparisons among sampled investigations. A preference is granted only when visual or retrieved evidence substantiates its underlying claims, preventing persuasive but unsupported content from being rewarded. Across four VAI and four VQA benchmarks, including our VAIBench, Vision-AutonoSearch-9B achieves the highest VAI average (66.2) and overall average (61.4) among the compared models, and exceeds three specialized search agents on both task averages. Ablations show that ECCR raises VAIBench from 69.9 to 73.6 and reduces the unsupported claim rate from 10.6% to 7.3%. Adding VAI data to VQA training under a matched budget improves all three VQA benchmarks, suggesting that learning to investigate strengthens capabilities shared with question answering.

Primary area:
foundation or frontier models, including LLMs
Keywords:
autonomous visual investigation, multimodal search agents, vision-language models, reinforcement learning, evidence-grounded reasoning, explanation evaluation, tool-use agents, group relative policy optimization, image interpretation, multimodal LLM

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_PLeyOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes the Visual Autonomous Investigation (VAI) task, in which an agent identifies the contextual meaning of an image and explores external information to provide explanations. Also, the authors propose Vision-AutonoSearch (VAS), a framework to address this task.

Critical strengths
  1. The authors propose a task called Visual Autonomous Investigation (VAI), where an agent independently selects what to investigate and uses external information (when given an image and a general description).
  2. The authors propose the Evidence-Constrained Comparative Reward (ECCR), which incorporates preference only when a core argument is substantiated by visual information or retrieved evidence, in addition to scores based on fixed evaluation criteria.
  3. The authors introduce Vision-AutonoSearch-9B that has been trained on VAI task. This model outperforms stronger models on the VAI task.
Critical weaknesses
  1. In my opinion, the task's novelty hasn't been fully explained. This paper defines VAI as a task where the agent determines its own investigation direction without being given a specific question to answer. However, if we had asked the search agent for a general request like "Explain this image" (like Figure 1), it's possible that we could have had similar autonomous investigations and explanations in the past. The paper compares existing agents, but doesn't directly compare their performance under the same general requests and search conditions, leaving unclear whether VAI demands new capabilities or simply applies new prompts and evaluation settings to existing search agents. To compare the VAS with existing agents, a controlled experiment is needed, using the same inputs, tools, and budget.
  2. I believe a more detailed explanation of the judge model is required. The authors verify alignment with humans in-house for only 20 responses. This is not sufficient to fully verify the judge's reliability.
  3. The authors used GPT-6 for dataset building, but why are the experiments conducted with GPT-5?
  4. Overall, I believe that existing agents have already been handling this task, even though it wasn't explicitly defined before. I think that's why the difference in score between the baseline and the Vision-AutonoSearch-9B isn't that significant. Overall, I believe this paper needs to be revised in terms of motivation, readability, and detail.
Questions for authors
  1. Why was Figure 1 placed on page 1? I think Section 4.2 above looks more better for readability.
  2. Which GPT-6 model was used for this work? (line 201) Also, in line 715, the author mentioned they used GPT-5.6-Sol, not GPT-6. Also, in the experiments, the author used GPT-5 (same, which GPT-5?). I was so confused about this.
  3. What happens if user ask like "Explain this image" the (strong) agent to describe this image in Figure 1? I think we can get similar response.
8 comments
AuthorsOpenReview ↗

Response to Reviewer PLey (Part 1/2)

Thank you for your detailed comments. We have clarified what VAI asks an agent to do, evaluated existing search agents under the same general request, expanded judge validation, and revised the figures and model descriptions.

W1 / W4 / Q3. Does VAI require more than a general image prompt?

We agree that a sufficiently strong general-purpose model can already investigate an image when asked to explain it. For example, GPT-5.6-Sol with our tools investigates the inscriptions in The Threatened Swan and explains their later political meaning using three sources. We now include this agentic frontier baseline in Table 1.

We emphasize that VAI asks the model to decide what to investigate and when the investigation is complete. In conventional search-augmented VQA, the user's question specifies the investigation target and provides a basis for judging whether the answer is sufficient. In VAI, the model must identify visual clues worth pursuing, choose an investigation direction, and then judge whether the available evidence supports a complete explanation.

We evaluated the specialized agents using the same request, “Help me understand this image,” with the same tools and budget. The following comparison on The Threatened Swan illustrates the different behaviors.

AgentSearch behaviorFinal report
Qwen3.5-9B, agentic backboneStops after 2 searchesIdentifies the painting and artist, but misses the inscription.
SearchEyes-9BRepeatedly searches the painting's title and subject, reaches the finalization notice after multiple searchesDescribes the painting but does not explain the inscription.
VAS-9BSearches the inscription by its third query and finishes after 4 searchesExplains the political allegory and the later-added inscriptions with 3 sources.

These trajectories show how the agents differ after identifying the painting, with VAS selecting the inscription as a further clue to explain. We provide the complete trajectories in Appendix G.1.

We also tested the same image with a specific question about the word “Holland” on the egg. The specialized agents then retrieved the relevant information, answered, and stopped. The contrast shows why choosing the investigation target matters. When the question supplies that target, the same search tools can find the answer.

We observe a similar pattern in our audit of 100 VAIBench images. Among the specialized-agent queries, 95% reformulate the main subject, compared with 21% for VAS-9B. Specialized agents investigate 1.1 distinct topics per image on average, compared with 2.8 for VAS. Their episodes reach the harness's finalization notice 82% of the time, compared with 9% for VAS. When given a specific question, the specialized agents' rate falls to 11%. Appendix F.2 reports the coding criteria and counts.

In the aggregate evaluation, the three specialized open baselines score 37.1 to 45.9 on VAI average. VAS-9B reaches 66.2 and outperforms them on all four VAI benchmarks. Within our training ablation, VAIBench rises from 67.1 with SFT to 69.9 with GRPO and 73.6 with ECCR. The trajectory analysis shows behavioral differences in choosing investigation targets and deciding when to stop, while the training ablation supports improvements in overall explanation quality. Together, these findings support dedicated training and evaluation for autonomous visual investigation.

W2. How reliable is the judge?

We expanded human validation from 20 responses to 200 responses covering all 100 VAIBench images, with two raters who did not know the models' identities or automatic scores. We also re-scored the responses with Claude Opus 5.5, an independent judge.

The human raters and both judges use the same criterion-level scoring rubric, assigning 0, 0.5, or 1 based on the report content and the evidence actually acquired by the agent.

ComparisonExact agreementWeighted κExtreme disagreements, 0 vs 1
Human rater vs human rater82%0.723.5%
DeepSeek vs adjudicated human consensus84%0.742.5%
Claude vs adjudicated human consensus87%0.791.8%
DeepSeek vs Claude87.5%0.792.5%

Both the independent judge and human consensus confirm ECCR's improvement. Claude measures a 3.1-point VAIBench gain and human consensus a 3.0-point gain, with a 3.0-percentage-point UCR reduction in both evaluations. We include the full agreement protocol and results in Appendix E.

AuthorsOpenReview ↗

Response to Reviewer PLey (Part 2/2)

W3 / Q2. Which GPT models did we use?

Previous labelExact modelRole
GPT-6GPT-6.1-SolDrafts reference explanations and rubrics for VAI-RL-2K.
GPT-5.6-SolGPT-5.6-SolReviews VAIBench annotations alongside DeepSeek-V4-Flash.
GPT-5GPT-5.6-SolServes as a Table 1 baseline, now evaluated with and without tools.

We used GPT-6.1-Sol to help produce high-quality reference explanations and rubric drafts, and selected GPT-5.6-Sol as a frontier-model baseline. In revised Table 1, we also evaluate GPT-6.1-Sol with the same tools as VAS.

We have standardized the model names throughout the paper and clarified their roles in Appendix D.5. We also corrected the model name in Section 3.1 and Appendix A.1 to reflect that the SFT demonstrations were generated using GLM-5.3-Flash.

Q1. Figure placement and readability

Thank you for this constructive layout suggestion. We have moved the results overview to Figure 3 in Section 4.2 beside Table 1 to improve the flow of the experimental section. We use the new Figure 1 to introduce the task through The Threatened Swan, and have shortened the opening explanation to make the motivation easier to follow.

We sincerely appreciate your feedback. We hope the new controlled experiments, case study, and clearer presentation address your concerns, and we would be grateful if you considered updating your score.

Thanks to the authors for their response. However, the authors mention Appendix D.5, Appendix E, ..., in their response, but there are no corresponding Appendix sections. I think the authors need to check it again. Please let me know after resolving this issue.

Reviewer_6h1vOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces new task called VAI. In this task, an agent is given an image and a general prompt ("Explain this image") and identify visual clues, search for external information and synthesize explanation autonomously. While in the standard search-augmented VQA, the query explicitly directs the search. The core contribution is the Evidence-Constrained Comparative Reward for GRPO training, which augments binary rubric scores with sparse pairwise comparisons, with the support by VAI-SFT-5K and VAI-RL-2K training datasets and VAIBench for evaluation.

Critical strengths
  1. The paper moves from VQA which directs the search direction in query to Visual Autonomous Investigation that drives the agent to autonomously search for clues and search for external information. In Table 2, the ablation studies reveal that training only on VAI with GRPO transfers to VQA tasks. This demonstrates learning what to investigate also strengthens general reasoning.
  2. RL rewards for open-ended reports suffers from dilemma: fixed rubrics lack granularity and LLM-as-a-Judge pairwise preferences are easily hacked. The ECCR solves this by combining rubric comparative score which strictly requires retrieved evidence verification.
Critical weaknesses
  1. DeepSeek-V4-Flash is used for the rubric judge, pairwise judge in ECCR, the claim auditor for UCR and the judge model in VAI benchmarks. Optimizing a policy against a judge model and evaluating test performance with the exact same judge model introduces the risk of judge stylistic overfitting. Replace the judge model with an independent model in the final evaluation is necessary to verify that the reported gains are robust.
  2. In table 1, GPT-5, Gemini-2.5-Pro, Claude-4-Sonnet are evaluated in "Direct Reasoning" mode without search tools, whereas VAS-9B operates in an agentic workflow. Lacking a zero-shot agentic baseline using frontier models entangles the difficulty of the benchmark and the advantage of search tool access.
Questions for authors
  1. Could you report the evaluation results on VAIBench using an independent judge model from a different family? This can make the experiments results more convictive.
  2. How do frontier models (GPT-5, Gemini-2.5-Pro, Claude-4-Sonnet) perform on VAIBench when equipped with the exact same search tools?
  3. In Table 1, search-augmented agents fall behind direct-reasoning GPT-5 obviously on MemeBench. What causes agentic workflows to struggle on this benchmark?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 6h1v (Part 1/2)

Thank you for recognizing the VAI formulation, the VAI-to-VQA transfer, and ECCR. We have added same-tool frontier baselines and independent judge results, and strengthened our MemeBench analysis.

W1 / Q1. Independent evaluation of VAIBench

We used Claude Opus 5.5 to independently score the saved VAIBench outputs of all 19 model settings in the revised Table 1. Claude received the same scoring rubric and the evidence observed by each agent.

ModelSettingDeepSeek-V4-FlashClaude Opus 5.5Claude − DeepSeekClaude rank
GPT-5.6-SolDirect Answer34.636.4+1.817
GPT-6.1-SolDirect Answer30.737.2+6.516
Gemini-2.5-FlashDirect Answer27.921.9−6.018
Gemini-2.5-ProDirect Answer43.343.9+0.612
Gemini-3.1-ProDirect Answer37.237.5+0.315
Claude-4-SonnetDirect Answer18.018.6+0.619
GPT-5.6-SolAgentic Workflow80.774.6−6.15
GPT-6.1-SolAgentic Workflow84.681.8−2.81
Gemini-2.5-ProAgentic Workflow77.676.2−1.44
Gemini-3.1-ProAgentic Workflow81.680.2−1.42
Claude-4-SonnetAgentic Workflow64.260.0−4.28
Kimi-K2.5Agentic Workflow71.765.8−5.97
Qwen3.5-9BAgentic Workflow43.042.6−0.413
Qwen3.5-35B-A3BAgentic Workflow49.545.4−4.111
OpenSearch-VL-8BSpecialized Search57.752.4−5.39
Vision-DeepResearch-8BSpecialized Search43.742.6−1.113
SearchEyes-9BSpecialized Search53.246.0−7.210
VAS-9BSpecialized Search73.669.0−4.66
VAS-35B-A3BSpecialized Search79.578.1−1.43

The two judges show Kendall τ-b = 0.95 across the 19 settings. Tied Claude scores share a rank. Claude also finds that ECCR improves VAIBench by 3.1 points, increases Q by 1.8 and E by 4.4, and lowers UCR by 3.0 percentage points relative to joint GRPO. Two blinded human raters independently confirm these gains on 200 responses. We provide the full judge and human agreement analysis in Appendix E.

W2 / Q2. Frontier models with the same search tools

We added agentic evaluations of all three models in your question, GPT-5.6-Sol (labeled GPT-5 in our submission), Gemini-2.5-Pro, and Claude-4-Sonnet, together with the newer GPT-6.1-Sol and Gemini-3.1-Pro. They use the same three tools, general image request, harness, and budget as VAS.

ModelSettingVAI AvgVAIBenchMemeBench
GPT-5.6-SolDirect Answer61.934.669.8
GPT-5.6-SolAgentic Workflow76.680.769.2
GPT-6.1-SolDirect Answer64.530.759.6
GPT-6.1-SolAgentic Workflow83.284.673.9
Gemini-3.1-ProDirect Answer67.937.269.2
Gemini-3.1-ProAgentic Workflow80.481.671.7
Gemini-2.5-ProDirect Answer57.443.355.3
Gemini-2.5-ProAgentic Workflow68.377.659.9
Claude-4-SonnetDirect Answer43.018.042.2
Claude-4-SonnetAgentic Workflow57.264.244.7
OpenSearch-VL-8BSpecialized Search45.957.727.3
SearchEyes-9BSpecialized Search43.453.225.6
Vision-DeepResearch-8BSpecialized Search37.143.719.9
VAS-9BSpecialized Search66.273.649.8
VAS-35B-A3BSpecialized Search73.679.562.9

These results show the value of tool-enabled investigation. GPT-5.6-Sol improves its VAI average by 14.7 points with tools, and Gemini-2.5-Pro and Claude-4-Sonnet improve theirs by 10.9 and 14.2 points. Its VAIBench score rises by 46.1 points; the explanation-quality component alone rises by 14.8 points, separate from the rubric's zero evidence score for answers without retrieved sources.

They also show the value of VAI-specific training. Existing specialized open search agents reach 37.1 to 45.9 on VAI average, compared with 66.2 for VAS-9B. Our VAS-35B-A3B reaches 73.6, within 3.0 points of GPT-5.6-Sol with tools, and scores 79.5 on VAIBench compared with GPT-5.6-Sol's 80.7. GPT-6.1-Sol remains the strongest tested agent at 83.2 VAI average. We now show these comparisons directly in Table 1.

AuthorsOpenReview ↗

Response to Reviewer 6h1v (Part 2/2)

Q3. Why do specialized agents perform worse than direct GPT on MemeBench?

The specialized search agents and direct-answer GPT in the original table differ in their backbones and training. To better understand this gap, we added comparisons of the same model with and without tools. GPT-6.1-Sol improves on MemeBench from 59.6 to 73.9, Gemini-3.1-Pro improves from 69.2 to 71.7, Gemini-2.5-Pro from 55.3 to 59.9, Claude-4-Sonnet from 42.2 to 44.7, and GPT-5.6-Sol remains nearly unchanged at 69.8 versus 69.2. These results show that models with the relevant image-understanding capabilities can use search to further improve their meme explanations.

To understand why specialized search agents still score lower, we examined their low-scoring reports and frequently found incorrect character identification and missed cultural references. These errors affect the choice of investigation targets. Interpreting a meme often requires connecting specific characters and composition with cultural context to formulate useful queries and assess how the retrieved information relates to the joke. Even after retrieving the correct template origin or background facts, the model must connect that information to the particular meme. These observed errors help explain the gap between specialized search agents and frontier models. We include representative cases in Appendix G.3.

We also compare performance before and after specialized training within the same agentic workflow to assess how much training can improve this task. VAS-9B increases MemeBench from 19.9 to 49.8 over Qwen3.5-9B, while VAS-35B-A3B increases it from 26.3 to 62.9 over Qwen3.5-35B-A3B. These are gains of 29.9 and 36.6 points, respectively. The results show that investigation-specific training substantially improves meme explanations under a fixed search workflow.

Thank you again for your support of our work. We hope the additional independent evaluations, controlled comparisons, and error analysis address your questions and strengthen your confidence in the paper. We would be grateful if you considered a stronger recommendation for acceptance in light of this additional evidence.

Dear Reviewer 6h1v,

Thank you for your constructive comments. We would very much appreciate your comments on whether our rebuttal addresses your questions and comments.

Sincerely,

The authors

Dear Reviewer 6h1v,

Thank you for your thoughtful review and for recognizing the value of Visual Autonomous Investigation and ECCR. As the discussion period draws to a close, we would be happy to answer any remaining questions about the independent evaluation, the frontier models tested with the same tools, or the MemeBench analysis. If these additions address your concerns, we would appreciate a brief note letting us know.

Thank you for your time.

Best regards,

The authors

Reviewer_usDLOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Visual Autonomous Investigation (VAI), an agent automatically judge what needs investigation, retrieves relevant evidence and produces a grounded explanation, where model only receives an image and a general request. The authors present Vision-AutonoSearch(VAS) through SFT and joint RL on VAI and search-augmented VQA. This paper's core method is Evidence-Constained Comparative Reward(ECCR) which has image-specific rubric scores with pairwise comparisons between sampled investigations. VAS gets highest VAI and overall average scores. ECCR improve VAIbench from 69.9 to 73.6, evidence-grounded completion improve 65.5 to 70.3. unsupported claim rate (UCR) reduced from 10.6% to 7.3%.

Critical strengths
  1. This task has practical implication. Users often want to understand an image but do not know how to propose questions.
  2. The reward design is reasonable. Each candidate has its own evidence packet and offer abstention option.
  3. Training ablation provides useful controlled comparison. RL ablations have the same setup such as SFT initialization. The comparison between joint rubric-only GRPO and joint GRPO with ECCR has VQA supervision fixed.
Critical weaknesses
  1. Main benchmarks have no consistent experiment conditions. Published baseline results retain their source protocols, which may differ in data subsets, tools, and judges.
  2. The ablation does not isolate the central evidence constraint. Table 2 compares rubric-only GRPO with the full ECCR package. This simultaneously introduces additional pairwise feedback and the particular requirement that decisive claims be supported by evidence. The improvement therefore does not establish which component is responsible.
  3. Evaluation is insufficiently independent of the training judge. The same DeepSeek-V4-Flash model supplies training rewards, final benchmark scores, unsupported-claim audits, and visit summaries. VAIBench annotations are also retained only when this model and GPT-5.6-Sol agree on their acceptability. This supports internal consistency, but leaves open whether training improves independently assessed quality or increasingly satisfies the preferences of the shared evaluator.
Questions for authors
  1. Which Table 1 results were newly evaluated, and which were imported?
  2. Does the evidence constraint itself improve performance? How does ECCR compare with an ordinary pairwise reward and an evidence-aware pointwise reward when judgment effort and actor training budgets are matched?
  3. Do the improvements survive independent evaluation? Can a different judge family and blinded human raters reproduce the gains in explanation quality, evidence support, and unsupported-claim rate?
2 comments
AuthorsOpenReview ↗

Response to Reviewer usDL

Thank you for recognizing the value of VAI and our evidence-constrained reward design. To address your questions, we have added more rigorous controlled comparisons and independent evaluations.

W1 / Q1. Which results were evaluated by us?

We evaluated all 40 VAI results and 32 of the 40 VQA results in the submitted Table 1. Only eight VQA results were taken from the source papers. In the revised table, we evaluated all 76 VAI results and 68 of the 76 VQA results. We mark the other eight VQA results with a dagger and identify their sources.

ResultsSubmitted tableRevised tableEvaluation protocol
VAI, evaluated by us4076The same image request, instructions, scoring, and harness. Tool-enabled models share the same tools and budget.
VQA, evaluated by us3268Our evaluation environment with each benchmark's original questions and scoring criteria.
VQA, reported by prior work88Source-paper results, clearly marked as such.

For VAI, every model receives the request “Help me understand this image.” Tool-enabled models use the same text_search, visual_search, and visit tools. We allow a 128K context, at most 32 model calls, 60 tool calls with up to four in parallel, 8,192 output tokens per call, 32,768 cumulative output tokens, and 30 minutes. Direct Answer models do not receive tools.

We now separate Direct Answer, Agentic Workflow, and Specialized Multimodal Search Agents in Table 1. The VAI comparison among tool-enabled agents is conducted under our common protocol.

W2 / Q2. Does the evidence constraint improve ECCR?

We added GRPO + Pair and GRPO + EvPoint to separate evidence verification from pairwise feedback. All variants start from the same Qwen3.5-9B SFT checkpoint and use the same training data and actor budget. Each new variant adds eight judge calls per rollout group, matching ECCR's number of calls.

Pair retains ECCR's pair selection, evidence packets, and comparative weight, but removes verification of the decisive claim. EvPoint scores each response individually for claims supported by its own retrieved evidence, without pairwise comparison.

RewardAdditional judge calls per groupVAIBenchQ ↑E ↑UCR ↓
GRPO069.974.365.510.6%
GRPO + Pair871.476.266.610.8%
GRPO + EvPoint872.175.468.88.6%
GRPO + ECCR873.676.970.37.3%

Compared with Pair, ECCR gains 2.2 VAIBench points, improves evidence support by 3.7 points, and reduces UCR by 3.5 percentage points. Compared with EvPoint, ECCR gains another 1.5 VAIBench points and reduces UCR by 1.3 percentage points.

These results identify the contribution of evidence verification. Pairwise feedback without verification leaves UCR at 10.8%, while both evidence-aware rewards lower it. ECCR achieves the best explanation quality and evidence support.

Claude Opus 5.5 confirms the pattern. Under this independent judge, ECCR scores 69.0, compared with 66.5 for Pair and 68.2 for EvPoint. Its UCR is 10.0%, compared with 13.6% and 11.2%, respectively. We added both ablations and the full cross-judge results to Section 4.3 and Appendix E.

W3 / Q3. Do the gains hold with an independent judge and human raters?

We re-scored the same saved outputs from joint GRPO and ECCR with Claude Opus 5.5, which was not involved in data construction, annotation review, or training. We also repeated the unsupported-claim audit. Two human raters independently evaluated all 200 responses from 100 VAIBench images, without seeing model identities or judge scores. They resolved disagreements after completing their independent ratings.

MetricDeepSeek-V4-FlashClaude Opus 5.5Human consensus
Q ↑74.3 → 76.9 (+2.6)71.3 → 73.1 (+1.8)72.0 → 74.0 (+2.0)
E ↑65.5 → 70.3 (+4.8)60.5 → 64.9 (+4.4)61.0 → 65.0 (+4.0)
VAIBench ↑69.9 → 73.6 (+3.7)65.9 → 69.0 (+3.1)66.5 → 69.5 (+3.0)
UCR ↓10.6% → 7.3% (−3.3 pp)13.0% → 10.0% (−3.0 pp)12.5% → 9.5% (−3.0 pp)

All three evaluations agree on the direction of improvement. In particular, Claude measures a 3.1-point VAIBench gain and human consensus measures 3.0 points. Both find a 3.0-percentage-point reduction in UCR.

The human raters reached 82% exact agreement before discussion, with weighted κ 0.72. Against their adjudicated consensus, DeepSeek achieved 84% agreement and κ 0.74, while Claude achieved 87% agreement and κ 0.79. Claude also re-scored all seven Table 2 training configurations, and we report the complete results in Appendix E.

We sincerely hope these additional controls and independent results address your concerns and would appreciate your reconsideration of our score.

Dear Reviewer usDL,

Thank you for your constructive comments. We would very much appreciate your comments on whether our rebuttal addresses your questions and comments.

Sincerely,

The authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response by Authors

We thank all three reviewers for their careful reading and for recognizing the value of VAI and evidence-constrained rewards. Our goal is to train agents to identify visual clues worth investigating, acquire supporting evidence, and judge when they have a sufficient explanation.

We evaluated existing specialized multimodal search agents under the same open-ended request, tools, and budget. Vision-AutonoSearch-9B outperforms these agents on all four VAI benchmarks, with a VAI average of 66.2, compared with 37.1–45.9 for the specialized baselines, while remaining competitive on search-augmented VQA. We also added frontier-model baselines with the same tools and budget.

To examine ECCR's evidence constraint, we added GRPO + Pair and GRPO + EvPoint, using the same SFT initialization, training data, actor budget, and number of additional judge calls as ECCR. Pair removes verification of the decisive claim, and EvPoint replaces comparisons with evidence-aware pointwise scoring. ECCR improves VAIBench over these controls by 2.2 and 1.5 points, respectively, and achieves the lowest unsupported-claim rate of 7.3%.

We independently evaluated the same saved outputs with Claude Opus 5.5, which was not involved in data construction or training, and two blinded human raters who scored 200 responses covering all 100 VAIBench images. Claude and human consensus measure ECCR gains over joint GRPO of 3.1 and 3.0 VAIBench points, respectively, together with improved explanation quality and evidence support. Both also measure a 3.0-percentage-point reduction in unsupported claims. We additionally re-scored all 19 model settings in Table 1 with Claude, obtaining Kendall τ-b = 0.95 between the two judges' rankings.

We have also clarified model names and roles, and revised the figures and organization to make the results and their supporting evidence easier to follow. We sincerely hope this additional evidence addresses the reviewers' concerns and strengthens confidence in our contribution and experimental findings.

We have released the model checkpoints and evaluation instructions for Vision-AutonoSearch-9B and Vision-AutonoSearch-35B-A3B through an anonymous account.

We have updated the manuscript accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.