DIAGNOSE WHAT YOU DESCRIBE: ALIGNING VISUAL EVIDENCE WITH MAMMOGRAPHY DIAGNOSIS

Reliable predictions require agreement between outputs and supporting evidence, which is essential in high-stakes settings. However, recent Vision–Language Models often produce predictions that are not supported by the evidence they present. This issue becomes critical in medical imaging, where such inconsistencies can undermine trust in automated systems. In mammography diagnosis, for example, a model may predict malignancy while describing features that are typically associated with benign findings. To address this problem, we propose a Knowlwdge-Driven Reasoning (KDR) framework. The model generates multiple candidate predictions for each image, each paired with its supporting evidence. We evaluate each candidate by checking whether the evidence is consistent with the predicted diagnosis, which allows us to identify and discard unsupported outputs. The final prediction is selected based on this consistency criterion. A metric to measure agreement between evidence and diagnosis is introduces, which complements standard accuracy. Experiments on multiple public datasets show that the proposed framework improves classification performance and consistency.