A Training-Free Trustworthy Framework for Medical Image Diagnosis with Large Language Models

Large language models (LLMs) enable training-free and concept-level interpretable medical image diagnosis. However, two issues prevent their clinical use: hallucination, under which erroneous predictions carry nearly the same high confidence as correct ones and caused the model untrustworthy, and the long-tailed class distribution of medical data, which makes common diseases overwhelm rare ones. To address these issues, in this paper, we propose a novel trustworthy diagnosis framework built entirely on frozen LLMs without any training. To achieve the trustworthy diagnosis, we design two inference pathways: a concept matching pathway that diagnoses based on medical concepts, and a visual perception pathway that diagnoses from visual features. Both pathways are further corrected by a long-tailed prior debiasing module to tackle the long-tailed distribution issue. Then, a Dirichlet evidence fusion module finally converts the two debiased posteriors into evidential concentrations, producing a trustworthy fused diagnosis together with a sample-wise uncertainty. Experiments on benchmark datasets show its effectiveness. Besides, the uncertainty of our method is aligned with the accuracy of the prediction, enabling hallucination detection that traditional confidence alone cannot provide.