- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper targets a weakness of retrieval-augmented KB-VQA: retrieved evidence itself is often unreliable, yet existing training-free methods only handle knowledge conflict during decoding, so noisy evidence still propagates into reasoning. CAER proposes a training-free framework that uses detected conflict as an actionable feedback signal to refine retrieved evidence before answer generation. It comprises three stages: entity-aware evidence acquisition, conflict-aware evidence refinement and reliability modeling, and reliability-guided knowledge fusion. The main contribution is the conflict-evidence feedback mechanism and the explicit reliability-guided fusion, with a careful control attributing gains to the mechanism.
Critical strengths
- Relative to concurrent conflict-aware decoding methods, CAER propagates detected conflicts back to the evidence layer to drive conflict-focused query rewriting and in-pool reranking, and explicitly estimates evidence reliability to guide trust fusion. This is a crisp, reproducible, adoptable feedback loop, not engineering assembly.
- The authors re-run the strongest training-free baseline under fully matched settings and win on both metrics, ruling out scoring bias; then they run control experiments showing the gain comes from the conflict-refinement mechanism itself, not from extra compute or extra prompting.
Critical weaknesses
- every result is a single-run number with no variance or significance test, yet the ablation gaps are only ~0.4–1 point on a one-run 10K subset, likely within random noise. If those gaps don't survive multiple seeds, the "every component contributes" claim collapses; the hand-set weights also raise validation-overfitting concerns.
- only one conflict-aware baseline (CC-VQA) is actually benchmarked — the formally published CAD/AdaCAD/CoCoA are only mentioned in related work, with no explanation of why they're excluded.
Questions for authors
- All results in Tables 1–2 and 5–7 appear to be single-run point estimates. Can you provide mean ± std over ≥3 seeds and paired significance tests for the main results and, in particular, for the per-component ablation gaps (−0.38 to −0.96 on the 10K subset)?
- The weights α, β, ε, γ and the trust-fusion weights are fixed as single-config on a validation probe (Table 4), with only θ swept. Please provide sensitivity/grid analyses for at least α, β, ε, γ, and state whether these values would need re-tuning on another backbone or knowledge base.
- The formally published CAD, AdaCAD, and CoCoA are discussed only in Related Work and are not benchmarked under the KB-VQA protocol. Are they adaptable to this protocol, and if not, what specific obstacle prevents it?