CAER: Conflict-Aware Evidence Refinement for Knowledge-Based Visual Question Answering

Knowledge-based Visual Question Answering (KB-VQA) requires models to jointly understand visual content and retrieve external knowledge for factual reasoning. Existing retrieval-augmented methods typically follow a retrieve-then-generate paradigm, while recent training-free approaches further address conflicts between retrieved knowledge and parametric knowledge through conflict-aware decoding. However, these methods primarily treat knowledge conflict as generation-stage uncertainty, without exploiting it to refine retrieved evidence. Consequently, noisy knowledge and entity mismatch may still propagate into subsequent reasoning, limiting the effective utilization of external knowledge and degrading factual accuracy. In this paper, we propose CAER (Conflict-Aware Evidence Refinement), a training-free framework for KB-VQA that leverages knowledge conflict as feedback for evidence refinement. Specifically, CAER first performs entity-aware evidence acquisition to alleviate entity mismatch among retrieved candidates. It then conducts conflict-aware evidence refinement within a fixed candidate pool by exploiting detected conflicts to optimize evidence selection. Finally, CAER estimates evidence reliability by jointly modeling question relevance, entity consistency, visual alignment, and conflict consistency, and propagates the resulting reliability signal to a trust-guided knowledge fusion module that dynamically bal- ances retrieved evidence and parametric knowledge during answer generation. On InfoSeek, CAER achieves 48.2% accuracy while remaining fully training-free, outperforming the strongest training-free baseline by 3.58 percentage points. These results demonstrate the effectiveness of conflict-aware evidence refinement for improving factual reasoning in knowledge-based visual question answering.