acceptodds
Under review as a conference paper at ICLR 2027

CAER: Conflict-Aware Evidence Refinement for Knowledge-Based Visual Question Answering

Abstract

Knowledge-based Visual Question Answering (KB-VQA) requires models to jointly understand visual content and retrieve external knowledge for factual reasoning. Existing retrieval-augmented methods typically follow a retrieve-then-generate paradigm, while recent training-free approaches further address conflicts between retrieved knowledge and parametric knowledge through conflict-aware decoding. However, these methods primarily treat knowledge conflict as generation-stage uncertainty, without exploiting it to refine retrieved evidence. Consequently, noisy knowledge and entity mismatch may still propagate into subsequent reasoning, limiting the effective utilization of external knowledge and degrading factual accuracy. In this paper, we propose CAER (Conflict-Aware Evidence Refinement), a training-free framework for KB-VQA that leverages knowledge conflict as feedback for evidence refinement. Specifically, CAER first performs entity-aware evidence acquisition to alleviate entity mismatch among retrieved candidates. It then conducts conflict-aware evidence refinement within a fixed candidate pool by exploiting detected conflicts to optimize evidence selection. Finally, CAER estimates evidence reliability by jointly modeling question relevance, entity consistency, visual alignment, and conflict consistency, and propagates the resulting reliability signal to a trust-guided knowledge fusion module that dynamically bal- ances retrieved evidence and parametric knowledge during answer generation. On InfoSeek, CAER achieves 48.2% accuracy while remaining fully training-free, outperforming the strongest training-free baseline by 3.58 percentage points. These results demonstrate the effectiveness of conflict-aware evidence refinement for improving factual reasoning in knowledge-based visual question answering.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Visual Question Answering, Knowledge Conflict, MLLM

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8ZYHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets a weakness of retrieval-augmented KB-VQA: retrieved evidence itself is often unreliable, yet existing training-free methods only handle knowledge conflict during decoding, so noisy evidence still propagates into reasoning. CAER proposes a training-free framework that uses detected conflict as an actionable feedback signal to refine retrieved evidence before answer generation. It comprises three stages: entity-aware evidence acquisition, conflict-aware evidence refinement and reliability modeling, and reliability-guided knowledge fusion. The main contribution is the conflict-evidence feedback mechanism and the explicit reliability-guided fusion, with a careful control attributing gains to the mechanism.

Critical strengths
  1. Relative to concurrent conflict-aware decoding methods, CAER propagates detected conflicts back to the evidence layer to drive conflict-focused query rewriting and in-pool reranking, and explicitly estimates evidence reliability to guide trust fusion. This is a crisp, reproducible, adoptable feedback loop, not engineering assembly.
  2. The authors re-run the strongest training-free baseline under fully matched settings and win on both metrics, ruling out scoring bias; then they run control experiments showing the gain comes from the conflict-refinement mechanism itself, not from extra compute or extra prompting.
Critical weaknesses
  1. every result is a single-run number with no variance or significance test, yet the ablation gaps are only ~0.4–1 point on a one-run 10K subset, likely within random noise. If those gaps don't survive multiple seeds, the "every component contributes" claim collapses; the hand-set weights also raise validation-overfitting concerns.
  2. only one conflict-aware baseline (CC-VQA) is actually benchmarked — the formally published CAD/AdaCAD/CoCoA are only mentioned in related work, with no explanation of why they're excluded.
Questions for authors
  1. All results in Tables 1–2 and 5–7 appear to be single-run point estimates. Can you provide mean ± std over ≥3 seeds and paired significance tests for the main results and, in particular, for the per-component ablation gaps (−0.38 to −0.96 on the 10K subset)?
  2. The weights α, β, ε, γ and the trust-fusion weights are fixed as single-config on a validation probe (Table 4), with only θ swept. Please provide sensitivity/grid analyses for at least α, β, ε, γ, and state whether these values would need re-tuning on another backbone or knowledge base.
  3. The formally published CAD, AdaCAD, and CoCoA are discussed only in Related Work and are not benchmarked under the KB-VQA protocol. Are they adaptable to this protocol, and if not, what specific obstacle prevents it?
Reviewer_9XgbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the potential propagation of conflicting external knowledge and entity mismatches in knowledge based visual question answering. It proposes CAER, a training free approach that measures knowledge conflicts and conducts-aware evidence refinement. It proposes an three-stage process including entity-aware evidence acquisition, conflict aware evidence refinement and reliability modeling, and reliability-guided knowledge fusion. The experimental study shows that, with the same backbone, retrieve, knowledge, and judge, CAER achieves better performance on the InfoSeek benchmark than the closest baseline, CC-VQA.

Critical strengths
  1. The approach is training free, which make it lightweighted and applicable.

  2. The proposed three stage process is well-designed, coherent, and technically sound.

  3. The experiment results seem to be promising in a controlled setting.

Critical weaknesses
  1. The paper doesn't establish sufficient evidence of the claimed knowledge conflicts and their impact to the reasoning. The paper also needs to justify that the proposed CAER is a principled solution to address the claimed conflicts.

  2. The proposed CAER relies on the same VLM to extract entities, measure conflicts, and generate draft answers. This makes its performance sensitive to the capability of the VLM. This can also reinforce the mistakes made by the model itself. Meanwhile, if the model is capable of extracting correct entity and identify conflicts, the capability could also benefit the decoding phase directly. Although the experimental study suggests the added value of the proposed three stage pipeline, the paper should clarify on this in a more principled way.

  3. The potential conflict can arise from different sources, such as incorrect extraction of entities, conflict between the model parametric knowledge and retrieved evidences, and discrepancy among retrieved passages. Although using a single score to summarize the conflict sounds computationally efficient, it may not have sufficient granularity to precisely capture the conflicts on different dimensions that require different corrections. Given that c plays an important role in reranking and knowledge fusion, the paper should justify its formulation.

  4. The paper should discuss the complexity and cost of the proposed CAER, including the additional VLM calls, reranking operations, and dual-stream decoding.

Questions for authors

Please see the listed weaknesses.

Reviewer_DPztOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

To address the issue in KB-VQA where retrieved evidence is often unreliable and prone to conflict with a model’s parametric knowledge, this paper proposes a fully training-free, conflict-aware evidence refinement framework, named CARE. Departing from handling conflicts passively only during the decoding phase, CAER transforms knowledge conflicts into proactive optimization signals. First, it employs entity-aware re-ranking to mitigate "entity drift" during visual retrieval. Second, upon detecting a conflict, it dynamically rewrites the query and re-ranks candidates within a fixed pool to refine the evidence. Finally, it constructs a multidimensional reliability score to dynamically balance external evidence against internal knowledge during decoding. CARE outperforms existing training-free baselines of similar architecture on both the InfoSeek and OK-VQA benchmarks.

Critical strengths
  1. A novel approach to the KB-VQA problem, shifting the handling of knowledge conflicts from passive defense to proactive refinement. The paper demonstrates the effectiveness of this method through experiments. A particularly compelling aspect of this approach is that it circumvents a common pitfall in existing KB-VQA methods, where poor retrieval quality inevitably leads to poor answer generation. Instead, by identifying conflicts prior to answer generation, the method rewrites queries and re-ranks the candidate pool, effectively purifying the input context at its source.

  2. Through comprehensive experiments, including the replication of SOTA methods. The paper shows that, under consistent experimental settings, CARE achieves significant performance improvements.

Critical weaknesses
  1. The paper lacks an evaluation of the method's efficiency. Although CARE is a training-free framework, completing a full question-answering cycle involves a complex process requiring at least four calls to a VLM, specifically for entity prediction, conflict detection and analysis, draft generation, and knowledge-fused decoding. However, the main text provides no efficiency metrics whatsoever, which significantly hinders an assessment of the method's practical utility.

  2. The method relies on a large number of hyperparameters (as seen in Equations 7 and 14), yet there is no analysis or experimental investigation regarding the selection of these parameters.

Questions for authors
  1. Please refer to the "Weaknesses" section for the primary issues.

  2. The method relies heavily on the VLM’s ability to accurately identify conflicts and generate RconfR_{conf}. Please analyze the extent to which performance degrades in the following scenarios: (a) when the retrieved evidence is actually consistent with the parametric knowledge, but the VLM incorrectly flags it as a conflict; and (b) when a conflict exists, but the VLM fails to detect it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.