Multimodal Evidence Reasoning for Interpretable Breast Ultrasound Video Diagnosis

Deep learning has shown promising performance in breast lesion diagnosis, yet most existing methods are developed based on representative images manually selected by radiologists, limiting their ability to automatically perceive and identify diagnostic information from continuous ultrasound videos acquired during clinical scanning. Current methods are primarily optimized for final diagnostic predictions, providing limited evidence for understanding how their decisions are reached. Furthermore, achieving clinical interpretability in diagnostic models often depends on semantic guidance from radiologist-authored diagnostic reports paired with ultrasound videos, yet such reports are generally unavailable before diagnostic inference. To address these challenges, we propose a multimodal evidence reasoning framework for interpretable breast ultrasound video diagnosis. In clinical scenarios where diagnostic reports are unavailable prior to inference, we show through Fisher information analysis that aligning unpaired clinical text and ultrasound videos within a shared representation space can strengthen visual evidence representation of concepts. To reason over raw ultrasound videos, we further develop an evidence driven fusion strategy equipped with a dual-aware module to identify diagnostically informative frames, aggregate reliable evidence across time, and derive interpretable textual evidence. The resulting visual and textual evidence are subsequently integrated into multimodal explainable diagnostic chains, providing traceable support for the final diagnostic prediction. Extensive experiments on the large-scale BUS-VD breast ultrasound video dataset validate the effectiveness of the proposed framework in both diagnostic performance and interpretability. The source code is provided in the supplementary material.