- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes U-TRACE, a framework for open-domain VQA that first checks whether the VLM's initial answer is reliable and gathers more evidence only when it is not. To do this, the framework samples the VLM under a few different prompts to build an uncertainty profile. A detector uses this profile to flag likely-wrong answers. For flagged questions, a trained LLM agent decides step by step which tool to call (e.g., captioning or retrieval) and when to stop. The agent is trained with execution-verified learning: it runs each candidate tool from the same state and learns from whichever one improves the final answer. Experiments on OK-VQA and A-OKVQA show that U-TRACE improves performance across five base VLMs by 8–19 points and outperforms prior open-domain VQA methods.
Critical strengths
- The preliminary study is insightful. It shows that uncertainty can identify wrong answers but cannot tell which tool will fix them. These findings justify the proposed technique well.
- Execution-verified learning compares the outcomes of different tools from the same state, so the supervision is more reliable than training only on successful trajectories. The ablation study supports this.
- The authors evaluate extensively, covering five VLMs and two main benchmarks (plus transfer to VQAv2 and GQA in the appendix), and the results are promising.
Critical weaknesses
- No fine-tuned baseline: ToolVLM, ProbeDetector and RoleAgent are all LoRA-trained on the in-domain training split (Appendix E.1). The base VLM is only few-shot prompted. Without a baseline that fine-tunes the VLM on the same data, it is unclear how much of the gain comes from adaptive evidence acquisition and how much from in-domain supervision.
- Every answer passes through a final LLM rephraser with few-shot examples. Unflagged questions, which get no tool calls, still gain up to 5.1 points (Table 11). The authors attribute this to probe-conditioned answer generation and rephrasing combined (Sec. 5.3), but they never separate the two. The ablation in Table 4 should include a "without rephrasing" variant.
- Average inference time alongside performance should be reported for U-TRACE and all baselines to show the performance–cost trade-off. Since every question needs about 40 VLM samples plus several LLM calls, the added cost could be a serious limitation for real-world deployment.
- The agent policy is trained for one specific VLM and does not transfer well to others. In Appendix D.14, the policy trained for Qwen2.5-VL scores 40.1 on GLM's flagged questions, compared with 50.9 for GLM's own policy. It scores 40.3 on Llama-3.2-Vision's flagged questions, compared with 49.4. Every new VLM therefore needs its own costly training (detector, tool adapter and agent), which limits practicality.
Questions for authors
Please refer to the Weaknesses Section