acceptodds
Withdrawn from ICLR 2027

U-TRACE: Uncertainty-Guided Adaptive Evidence Acquisition for Open-Domain VQA

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Open-domain visual question answering requires systems to answer questions that may involve visual understanding, external knowledge, or multimodal reasoning. Existing approaches typically rely on fixed pipelines and single-pass generation, such as retrieval or visual grounding, without explicitly assessing whether the current answer is reliable or which intervention would improve it. To address these limitations, we introduce U-TRACE, an uncertainty-guided framework for adaptive evidence acquisition in open-domain VQA. U-TRACE probes the vision–language model under complementary visual and textual contexts to construct a model-specific behavioral state that captures semantic dispersion, context sensitivity and prediction stability. ProbeDetector uses this state to estimate the reliability deficit of the initial prediction and determine whether further evidence acquisition is required. For selected instances, a history-conditioned agent chooses the next evidence-seeking action and specifies its request when necessary. The resulting report is added to the evidence history and used to update the uncertainty state, so that each subsequent decision reflects what has already been observed. To learn the adaptive policy, we propose execution-verified learning, which compares alternative next actions from the same evidence history and assigns supervision based on their impact on final-answer quality. Experiments on OK-VQA and A-OKVQA show that U-TRACE consistently improves its base VLMs and outperforms existing open-domain VQA methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
agent, visual question answering, multimodality

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_LarUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes U-TRACE, a framework for open-domain VQA that first checks whether the VLM's initial answer is reliable and gathers more evidence only when it is not. To do this, the framework samples the VLM under a few different prompts to build an uncertainty profile. A detector uses this profile to flag likely-wrong answers. For flagged questions, a trained LLM agent decides step by step which tool to call (e.g., captioning or retrieval) and when to stop. The agent is trained with execution-verified learning: it runs each candidate tool from the same state and learns from whichever one improves the final answer. Experiments on OK-VQA and A-OKVQA show that U-TRACE improves performance across five base VLMs by 8–19 points and outperforms prior open-domain VQA methods.

Critical strengths
  • The preliminary study is insightful. It shows that uncertainty can identify wrong answers but cannot tell which tool will fix them. These findings justify the proposed technique well.
  • Execution-verified learning compares the outcomes of different tools from the same state, so the supervision is more reliable than training only on successful trajectories. The ablation study supports this.
  • The authors evaluate extensively, covering five VLMs and two main benchmarks (plus transfer to VQAv2 and GQA in the appendix), and the results are promising.
Critical weaknesses
  • No fine-tuned baseline: ToolVLM, ProbeDetector and RoleAgent are all LoRA-trained on the in-domain training split (Appendix E.1). The base VLM is only few-shot prompted. Without a baseline that fine-tunes the VLM on the same data, it is unclear how much of the gain comes from adaptive evidence acquisition and how much from in-domain supervision.
  • Every answer passes through a final LLM rephraser with few-shot examples. Unflagged questions, which get no tool calls, still gain up to 5.1 points (Table 11). The authors attribute this to probe-conditioned answer generation and rephrasing combined (Sec. 5.3), but they never separate the two. The ablation in Table 4 should include a "without rephrasing" variant.
  • Average inference time alongside performance should be reported for U-TRACE and all baselines to show the performance–cost trade-off. Since every question needs about 40 VLM samples plus several LLM calls, the added cost could be a serious limitation for real-world deployment.
  • The agent policy is trained for one specific VLM and does not transfer well to others. In Appendix D.14, the policy trained for Qwen2.5-VL scores 40.1 on GLM's flagged questions, compared with 50.9 for GLM's own policy. It scores 40.3 on Llama-3.2-Vision's flagged questions, compared with 49.4. Every new VLM therefore needs its own costly training (detector, tool adapter and agent), which limits practicality.
Questions for authors

Please refer to the Weaknesses Section

Reviewer_LLEeOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper addresses the limitations of existing fixed-pipeline and single-pass generation approaches in open-domain Visual Question Answering (VQA), which often fail to assess the reliability of their predictions or determine which interventions could improve them. To overcome this, the authors propose U-TRACE, a uncertainty-guided framework designed for adaptive evidence acquisition in open-domain VQA.

Critical strengths

None

Critical weaknesses

I strongly suspect that this manuscript is largely, if not entirely, generated by a Large Language Model (LLM). First, the task addressed in this paper should be properly defined as Knowledge-Based VQA (KB-VQA), a terminology shift that has been established in the community for some time, at least since my own prior work. Furthermore, although the "Related Work" section is titled "Open-Domain VQA," the content actually pertains to KB-VQA. This conflation leaves the reader entirely unclear as to what distinction, if any, exists between the two terms. Second, the logical flow of the Introduction is highly incoherent. It fails to clarify the precise definition of "uncertainty" in this context, nor does it explain how information filtering is actually performed. Regarding the evaluation, the authors fail to employ the most recent and widely adopted benchmark datasets. Finally, numerous aspects of the manuscript, ranging from the experimental setup to the writing details, reinforce my impression that it was directly generated by an LLM. I consider such a submission to be, to some extent, disrespectful to the reviewers, and as a result, I have assigned the lowest possible rating.

Questions for authors

None

Reviewer_PSXtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Today when a VLM provides answer for a VQA pair, we don't know if the model truly knows the fact or the answer is just from its guess. And even we know the model is guessing, we don't know which tool will help the model to get the correct answer. This paper aims to solve the above problem. It defines a multi-steps pipeline to tackle this problem: Test the models' stability -> filter out solid answers -> agent picks a tool and send VQA to the tool. During training, it also develops a execution verified learning which forces the agent to only call a tool if that tool can improve the final score.

Critical strengths
  1. Discover the Bad Evidence Trap. Most research on tool-using AI just assumes that giving a model more information always leads to better answers. This paper performs a diagnostic study proving that is not always the case. It shows we must teach models when the stop and ignore tools rather than just teaching how to use them.
  2. Teaching Agent through Real Consequence. This paper trains the agent based on the actual outcome of its actions. The training process compares what would happen if the model answered immediately versus what happens if it calls a tool. This is a practical training strategy which could easily be applied to other AI agents.
Critical weaknesses
  1. Self reflection vs External evidence. The paper's title is "Uncertainty-Guided Adaptive Evidence Acquisition for Open-Domain VQA" and put itself as a evidence acquisition system that fetches missing external information. However, looking through the whole paper, 4 of 5 tools do not fetch outside data at all. They just reprompt the same base VLM in a difference format (MULTIPLE CHOICE, CAPTION, INTERPRETATION, and FEW SHOT). True external search (RETRIEVAL) is barely used.
  2. Miss the compute match baseline. IIUC, for every question, U-TRACE generates 40 initial answers, feeds them into a ROLEAGENT, then runs a Final Rephraser to match VQA's exact-word grading. The baseline VLM only gets 1 generation and no rephrase. On unflagged questions where U-TRACE calls zero tools, accuracy still bumps up to +5.1 points. However, the paper never test a compute match baseline like a simple majority voting over these 40 initial samples and runs Final Rephraser. It is hard to tell how much of the gains comes from adaptive tools vs 40 sample emsembling and format cleanup.
  3. The HISTORYUQ Paradox. In Table 15, the authors prove the uncertainty after reading a tool report is a bad signal where it drops nearly identical rate for misleading reports and helpful reports. However, in Table 14 when they append this ut{hist}u_t^\{hist\} to the text history, accuracy on hard questions jumps by +3.2 to +5.0. It is unclear how the agent get +5.0 increase from a signal which is just proved unreliable.
Questions for authors
  1. Self reflection VS External evidence. Table 18 shows the RETRIEVAL is invoked only 4% - 10% while MULTIPLE CHOICE over existing candidates is invoked 29% - 37%. Does ROLEAGENT's heavy reliance on MULTIPLE CHOICE indicate that modern VLMs primarily need candidate arbitration rather than external evidence acquisition?
  2. There are several papers which also research the agentic pipeline for VQA. They first use agent to detect whether VQA instances need external context and if yes, the model will ask users for these context. Could you discuss how your framework fits alongside these works in Related Work section? example work: https://arxiv.org/abs/2507.13773, https://arxiv.org/abs/2601.16400, https://arxiv.org/abs/2603.07394

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.