acceptodds
Under review as a conference paper at ICLR 2027

Multimodal Evidence Reasoning for Interpretable Breast Ultrasound Video Diagnosis

Abstract

Deep learning has shown promising performance in breast lesion diagnosis, yet most existing methods are developed based on representative images manually selected by radiologists, limiting their ability to automatically perceive and identify diagnostic information from continuous ultrasound videos acquired during clinical scanning. Current methods are primarily optimized for final diagnostic predictions, providing limited evidence for understanding how their decisions are reached. Furthermore, achieving clinical interpretability in diagnostic models often depends on semantic guidance from radiologist-authored diagnostic reports paired with ultrasound videos, yet such reports are generally unavailable before diagnostic inference. To address these challenges, we propose a multimodal evidence reasoning framework for interpretable breast ultrasound video diagnosis. In clinical scenarios where diagnostic reports are unavailable prior to inference, we show through Fisher information analysis that aligning unpaired clinical text and ultrasound videos within a shared representation space can strengthen visual evidence representation of concepts. To reason over raw ultrasound videos, we further develop an evidence driven fusion strategy equipped with a dual-aware module to identify diagnostically informative frames, aggregate reliable evidence across time, and derive interpretable textual evidence. The resulting visual and textual evidence are subsequently integrated into multimodal explainable diagnostic chains, providing traceable support for the final diagnostic prediction. Extensive experiments on the large-scale BUS-VD breast ultrasound video dataset validate the effectiveness of the proposed framework in both diagnostic performance and interpretability. The source code is provided in the supplementary material.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodel, Interpretable model, Ultrasound video

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_pvqpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a multimodal evidence reasoning framework for interpretable breast ultrasound video diagnosis using unpaired clinical text and video evidence. The framework identifies key diagnostic frames, combines visual and textual evidence, and generates explainable diagnostic chains to support predictions. Extensive experiments on a large-scale breast ultrasound video dataset demonstrate its effectiveness in diagnostic performance and interpretability.

Critical strengths

This paper addresses the issue of explainability in breast ultrasound video diagnosis. It proposes a diagnostic framework that combines unpaired clinical text with reasoning based on video evidence. The proposed method can automatically extract key diagnostic information and provide both visual and textual explanations. It also shows potential for clinical use.

Critical weaknesses
  1. The frame-level weighted aggregation in Equation (10) is insensitive to the order of the input frames. It does not model temporal order or changes between frames. Therefore, the contribution of temporal information in continuous videos remains unclear.
  2. Shared projection parameters and improved downstream performance are not enough to demonstrate cross-modal semantic alignment. The proposed text-based semantic guidance without paired data lacks direct validation.
  3. The authors have shown that concept scores contribute to predictions, the roles of detection boxes, boundary maps, and text descriptions in the final predictions remain unclear. The current results do not fully demonstrate that the complete diagnostic process faithfully reflects the model's decision-making.
  4. The baselines include recent general video models, they cover only a limited range of methods.
  5. The paper describes a case-level data split but does not clarify whether each case corresponds to a unique patient. It also lacks key details about the data sources. Therefore, patient-level independence and experimental reproducibility remain difficult to assess.
  6. The experiments are conducted on only one private dataset, without independent external validation. The current results are not sufficient to support the model's ability to generalize across different data sources.
  7. The reference standard for benign and malignant labels is unclear. Agreement among doctors on these labels is also not reported.
Questions for authors
  1. The authors should explain what temporal information in continuous videos contributes compared with a set of independent ultrasound images. According to Equation (10), if each frame is encoded independently and then combined through weighted summation, simply changing the order of the input frames will not affect the final result.
  2. The authors should explain whether unpaired text truly achieves cross-modal semantic alignment. Sharing projection parameters does not necessarily mean that the two modalities are semantically aligned (Section 2.1).
  3. The authors should further clarify whether the proposed Multimodal Explainable Diagnostic Chain directly affects the final prediction or only provides an explanation after the prediction. For example, if key visual evidence is replaced or removed, or if concept descriptions and their corresponding scores are modified, will the final prediction change in a clinically meaningful way?
  4. The authors should include comparisons with more baseline methods proposed in recent years.
  5. The authors should clarify whether the training, validation, and test sets of the private dataset are independent at the patient level. More details about the data sources should also be provided. In addition, the authors should state whether the study received approval from the relevant ethics committee.
  6. The authors should report results on external datasets, such as [1] [2], to evaluate the generalization ability of the proposed method. Since Table 3 already compares the method proposed in [2], the authors should also explain why the dataset from [2] was not included in the external evaluation.
  7. The authors should explain how the labels were obtained. Specifically, they should clarify whether the labels were based on pathology results, clinical follow-up, or BI-RADS category mapping. An analysis of agreement among doctors on these labels should also be provided.

Ref [1] Clinical Knowledge Constrained Multi-Task Learning Framework for Breast Cancer Diagnosis Using Ultrasound Videos [2] A New Dataset and a Baseline Model for Breast Lesion Detection in Ultrasound Videos

6 comments
AuthorsOpenReview ↗

Response to Reviewer pvqp (1/4)

We sincerely appreciate your recognition of the clinical motivation and interpretability of our framework. Thank you for your constructive comments regarding video reasoning, cross-modal semantic alignment, diagnostic faithfulness, and dataset validation. We respond to your concerns below.

Response to Q1&W1

We acknowledge that Eq. (10) is permutation-invariant and does not explicitly model inter-frame temporal transitions. This design is motivated by the clinical characteristics of breast ultrasound examinations. During routine examinations, radiologists manually manipulate the probe to scan lesions from different angles and imaging planes. Consequently, the chronological order of frames is largely determined by the operator's scanning trajectory and technique, which may vary across examinations. As described in our Introduction, ultrasound videos capture lesion morphology from multiple probe angles and imaging planes, providing more comprehensive diagnostic information than individual images. Moreover, the same lesion may exhibit different characteristics across different imaging planes.

Accordingly, MIChain focuses on aggregating reliable diagnostic evidence across ultrasound frames rather than explicitly modeling acquisition order. Specifically, Eq. (8) extracts frame-level positive and negative evidence, together with evidential uncertainty Ut,cU_{t,c}. Eq. (9) further evaluates frame-level reliability by jointly considering evidential sufficiency and directional certainty, yielding the following reliability score:

Rt,c=∣et,c+−et,c−∣et,c++et,c−+2R_{t,c}=\frac{\lvert e_{t,c}^{+}-e_{t,c}^{-}\rvert}{e_{t,c}^{+}+e_{t,c}^{-}+2}

Based on this reliability, Eq. (10) computes concept-specific frame weights:

wt,c=exp⁡(Rt,c/η)∑t′=1Texp⁡(Rt′,c/η)w_{t,c}=\frac{\exp(R_{t,c}/\eta)}{\sum_{t'=1}^{T}\exp(R_{t',c}/\eta)}

The corresponding frame-level evidence is then aggregated as:

vc=∑t=1Twt,cet,c\mathbf{v}_c=\sum_{t=1}^{T}w_{t,c}\mathbf{e}_{t,c}

The ablation results in Table 5 demonstrate the effectiveness of evidence aggregation:

Table 5. Ablation studies of the key components in ourmodel.

Evidence AggregationGrading AccGrading KappaConcept AUPRConcept F1
Without84.3372.4556.4159.24
With87.1077.4660.9260.92
Improvement+2.77+5.01+4.51+1.68

Thus, MIChain adaptively aggregates diagnostic evidence across ultrasound frames according to its estimated reliability for each concept. As shown in Table 5, introducing evidence aggregation improves grading accuracy from 84.33% to 87.10% and Kappa from 72.45% to 77.46%, supporting the effectiveness of this design.

AuthorsOpenReview ↗

Response to Reviewer pvqp (2/4)

Response to Q2&W2

We clarify that MIChain promotes cross-modal semantic alignment by mapping visual and textual embeddings into a shared BI-RADS concept space, where both modalities jointly contribute to learning clinically meaningful concept representations through shared projection parameters and concept-level supervision.

1)Regarding the alignment mechanism, as described in Section 2.1, visual and textual embeddings are mapped into a shared concept space using the same learnable projection parameters:

ztvis=Whtvis+b,zjtxt=Whjtxt+bz_t^{\mathrm{vis}}=Wh_t^{\mathrm{vis}}+b,\qquad z_j^{\mathrm{txt}}=Wh_j^{\mathrm{txt}}+b

Importantly, the semantic correspondence is defined by the shared BI-RADS concept ontology: both modalities are represented over the same set of clinical concept slots, each associated with a corresponding BI-RADS description. Furthermore, visual concept supervision and textual supervision through Ltext\mathcal{L}_{\mathrm{text}} provide learning signals for these shared representations. Thus, the alignment is established at the clinical concept level, without requiring sample-wise video–text correspondence.

2)Regarding the theoretical basis, Eqs. (2)–(3) formulate visual and textual observations as independent observations of the same shared clinical concept factors θc\theta_c. Their Fisher information is therefore additive (Eq. (4)), i.e.,

Jv,t=Jv+JtJ_{v,t}=J_v+J_t

Theorems 1 and 2 further establish increased information and reduced estimation uncertainty under the stated assumptions (Eqs. (5)–(7)), providing a theoretical rationale for incorporating unpaired clinical text into shared concept representation learning.

3)To further validate the contribution of BI-RADS-specific semantics, we conduct additional controlled experiments using semantically meaningless text and medical text unrelated to breast lesions. We evaluate these alternatives under both the shared-projector-only and full evidence-guided video reasoning configurations. The results are summarized in Table 8.

Table 8. Ablation of textual content under different configurations (%).

Text ConditionUnpaired TextShared Proj.Evidence Reason.Evidence Aggr.Grading (Acc.)Grading (Kappa)Concept Prediction (AUPR)Concept Prediction (F1)
Shared Projector Only
No text×✓××83.4168.7358.9854.86
Meaningless text✓✓××82.9566.9458.2354.44
Irrelevant medical text✓✓××82.0366.9757.1354.02
BI-RADS text✓✓××84.7972.2259.3255.23
Full MIChain
No text×✓✓✓86.1875.1259.5658.96
Meaningless text✓✓✓✓85.7172.9659.2256.68
Irrelevant medical text✓✓✓✓83.8771.8758.1458.07
BI-RADS text✓✓✓✓87.1077.4660.9260.92

As shown in Table 8, BI-RADS descriptions consistently outperform both control text conditions across all four metrics under both configurations. Moreover, both control conditions perform below their corresponding no-text baselines. These results support the importance of BI-RADS-specific clinical semantics for shared concept representation learning, rather than merely introducing additional textual inputs. The complete experiments are provided in Appendix C.2 (Table 8), with a corresponding discussion in Section 3.3 of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer pvqp (3/4)

Response to Q3&W3

In MIChain, the final BI-RADS prediction is directly derived from the diagnostic evidence processed through the proposed Multimodal Explainable Diagnostic Chain, rather than being independently generated before the explanations. This is further demonstrated by the ablation studies of Evidence-Guided Video Reasoning and the concept intervention experiments.

1)Regarding the contribution of visual evidence to the final prediction, MIChain introduces Evidence-Guided Video Reasoning to extract and aggregate frame-level diagnostic evidence. Specifically, Dual-Aware Evidence Learning assesses the reliability of concept evidence within each frame, while Video Evidence Aggregation assigns concept-specific weights to emphasize frames providing more reliable evidence for different clinical concepts and suppress the influence of ambiguous or low-quality frames.

The ablation results in Table 5 demonstrate the contribution of these components:

ConfigurationEvidence LearningEvidence AggregationAcc.KappaAUPRF1
Without reasoning××84.7972.2259.3255.23
Evidence learning only✓×84.3372.4556.4159.24
Complete reasoning✓✓87.1077.4660.9260.92

These results demonstrate that identifying and prioritizing key visual evidence directly contributes to concept prediction and final grading, rather than merely providing post-hoc explanations.

2)Regarding whether modifying clinical concept scores can affect the final prediction, MIChain employs 14 clinical concepts derived from BI-RADS sonographic descriptors. Each predicted concept probability pcp_c is associated with its corresponding clinical description to provide interpretable textual evidence. More importantly, pcp_c directly gates the positive and negative evidence to construct the evidential concept bottleneck (Eq. (12)):

g=Concat⁡c=1C[pcvc+,(1−pc)vc−]g=\operatorname{Concat}_{c=1}^{C}[p_c v_c^{+},(1-p_c)v_c^{-}]

Following the test-time concept intervention paradigm of Concept Bottleneck Models (CBMs) [1], MIChain allows the predicted scores of clinical concepts to be modified and propagated to the final prediction. Specifically, intervention is performed by replacing the original concept probability pcp_c with a corrected value p~c\tilde{p}_c:

g~=Concat⁡c=1C[p~cvc+,(1−p~c)vc−]\tilde{g}=\operatorname{Concat}_{c=1}^{C}[\tilde{p}_c v_c^{+},(1-\tilde{p}_c)v_c^{-}]

The modified evidential concept bottleneck is then passed to the grading head:

y^gint=fgrade(g~)\hat{y}_g^{\mathrm{int}}=f_{\mathrm{grade}}(\tilde{g})

The existing intervention results in Fig. 4(b) demonstrate the impact of modifying concept scores on the final diagnostic performance:

TaskOriginal ScoresCorrected ScoresImprovement
Grading Accuracy87.1%89.4%+2.4 pp
Classification Accuracy88.9%89.8%+0.9 pp

Furthermore, Fig. 4(a) shows that MIChain consistently improves as the proportion of corrected concept scores increases, outperforming CBM and CEM across different intervention degrees. These results demonstrate that modifying the predicted scores of clinically meaningful concepts directly influences the final diagnostic prediction.

Reference

[1] Koh et al. Concept Bottleneck Models. International Conference on Machine Learning (ICML), 2020.

Response to Q4&W4

Following the reviewer's suggestion, we have expanded the baseline comparisons in Table 1 of the revised manuscript to include additional methods, particularly those proposed in 2025. Specifically, we have incorporated CVA-Net (MICCAI'22), CRL (MICCAI'25), and MVP-CBM (IJCAI'25). The corresponding results are summarized below.

Table 1. Additional baseline comparisons on BUS-VD (%).

MethodGrading AccGrading KappaGrading F1Classification AccClassification KappaClassification F1
CVA-Net (MICCAI'22)81.1168.6677.5582.4947.7073.37
CRL (MICCAI'25)79.7265.3260.1782.0351.6975.81
MVP-CBM (IJCAI'25)86.6473.1776.9788.0267.2283.56
MIChain (Ours)87.1077.4682.3888.9469.3784.62

As shown above, MIChain consistently outperforms these additional baselines across all six reported metrics for BI-RADS grading and benign/malignant classification. These results further demonstrate the effectiveness of MIChain compared with recent methods.

Reviewer_gTR8OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes MIChain, a multimodal interpretability chain for breast ultrasound video diagnosis. MIChain leverages unpaired clinical knowledge without relying on radiologist-authored reports, which are generally unavailable at inference time. Ultrasound videos and clinical text are projected into a cross-modal shared semantic space, followed by an evaluation mechanism for frame evidence reliability. Eventually, MIChain provides an explainable diagnostic chain for BI-RADS grading, reducing the gap between AI models and clinical diagnostic practice.

Critical strengths
  1. The paper explores a relevant task of interpretability in breast ultrasound video diagnosis, aiming to narrow the gap to radiologists' workflow. The final product includes visual and textual evidence on top of the video, in addition to a final BI-RADS grade, and may be useful for radiologists as a second-opinion tool.
  2. The authors provide a detailed mathematical formulation to explain the potential advantages of using unpaired video-text observations. Together with the empirical results, this supports the general setup of MIChain.
  3. The authors introduce BUS-VD, a large-scale private breast ultrasound video dataset. The benchmark contains 1,085 ultrasound videos, with clinical concepts annotated by five physicians and verified by an expert.
Critical weaknesses
  1. The related work section is missing from the main paper, and the discussion in the appendix omits several relevant prior works. The current related-work discussion does not establish what distinguishes MIChain from the closest existing approaches (see below). The authors should identify the closest prior works and distinguish their technical contribution from them in order to better establish their novelty.
  • Qiu et al. [1] and Yan et al. [2]: Existing interpretable video-based breast ultrasound diagnosis methods that provide BI-RADS-related diagnostic attributes. Similar to MIChain, [2] also selects keyframes and localizes lesions.

  • BI-RADS-Net-V2 [3]: An image-based breast ultrasound model that predicts BI-RADS descriptors and estimates their contributions to the final diagnosis.

  • MedCLIP [4]: Learns cross-modal representations from unpaired medical images and text, without requiring corresponding image-text pairs.

  1. The quantitative experiments show that MIChain outperforms generic interpretable and non-interpretable methods, but Table 1 lacks domain-specific breast ultrasound classification methods. KGA-Net [5] is a relevant prior method for video classification with keyframe-guided attention, although its incomplete public implementation may make numerical comparison impractical. CVA-Net [6], which includes a benign/malignant classification component and provides publicly available code and pretrained weights, is a reasonable candidate for numerical comparison. CVA-Net is already evaluated for lesion detection in Table 3, but not for benign/malignant classification. Please explain its exclusion from Table 1 if such a comparison is impractical.

Citations: [1] Qiu, Shunmin, et al. "Prospective assessment of breast lesions AI classification model based on ultrasound dynamic videos and ACR BI-RADS characteristics." Frontiers in Oncology 13 (2023): 1274557. [2] Yan, Pengfei, et al. "Clinical knowledge constrained multi-task learning framework for breast cancer diagnosis using ultrasound videos." Medical Image Analysis (2026): 104129. [3] Zhang, Boyu, Aleksandar Vakanski, and Min Xian. "BI-RADS-NET-V2: a composite multi-task neural network for computer-aided diagnosis of breast cancer in ultrasound images with semantic and quantitative explanations." IEEE access 11 (2023): 79480-79494. [4] Wang, Zifeng, et al. "Medclip: Contrastive learning from unpaired medical images and text." Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022. [5] Sun, Anlan, et al. "Boosting breast ultrasound video classification by the guidance of keyframe feature centers." International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer Nature Switzerland, 2023. [6] Lin, Zhi, et al. "A new dataset and a baseline model for breast lesion detection in ultrasound videos." International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer Nature Switzerland, 2022.

  1. The inference time of MIChain is significantly higher than all other interpretable methods. Please clarify how inference time is calculated and discuss the practical influence on using MIChain.

Minor weaknesses (visibility and readability issues):

  • Figure 2 is overloaded with details and hard to follow. Illustrating MIChain's general flow would be better for paper readability, while the full details already appear in the method section.
  • Please define NvN_v and NtN_t before Eq. 3 in the main paper.
Questions for authors
  1. Please clarify whether the baselines in Table 1 were trained from scratch or fine-tuned on BUS-VD, as well as the differences in evaluation protocol between the image-based and video-based models. The implementation details mention that all models were trained for 10K iterations. If this applies to all baselines - please clarify all the details: which layers were trained, full fine-tuning or LoRA, initial checkpoints, etc.
  2. How were the benign/malignant ground-truth labels in Table 1 obtained? Were they based on pathological diagnoses or derived from the BI-RADS grades? If the latter, please clarify the reported classification results, as BI-RADS categories do not necessarily correspond to confirmed benign or malignant diagnoses. Additionally, how is benign/malignant classification performed? Is it based on a separate classification head or derived from the predicted BI-RADS grades?
  3. Table 1 presents aggregate BI-RADS grading metrics, but according to Figure 6, the BI-RADS grades in BUS-VD are highly imbalanced. For example, only 20 out of 1,085 samples are Grade 5, potentially leaving ~4 Grade 5 samples in the test set is the split is equal. Due to the clinical importance of false positives and false negatives, per-grade precision/recall or a confusion matrix would be important.
8 comments
AuthorsOpenReview ↗

Response to Reviewer gTR8 (1/3)

We sincerely appreciate the reviewer's positive assessment and constructive feedback.

Response to W1

We appreciate the reviewer's suggestions regarding the related work and the novelty of MIChain. Following these suggestions, we have expanded the related work discussion in Appendix A.1 (Breast Ultrasound Diagnosis) and Appendix A.2 (Explainability in Medical Imaging) to incorporate the relevant studies and clarify the technical distinctions between MIChain and existing approaches. The revised content is provided below.

Appendix A.1 – Breast Ultrasound Diagnosis

Recent studies have incorporated clinical knowledge and BI-RADS-related attributes into breast ultrasound diagnosis. Qiu et al. (2023) extracted BI-RADS-related features from dynamic ultrasound videos for lesion assessment and classification, while Yan et al. (2026) further integrated keyframe selection and lesion localization for video-based diagnosis. At the image level, BI-RADS-Net-V2 (Zhang et al., 2023) predicts BI-RADS descriptors from ultrasound images and quantifies their contributions to the final diagnosis. Beyond clinical attribute-based diagnosis, MIChain learns shared concept representations from unpaired clinical descriptions and ultrasound videos, and performs reliability-aware reasoning over positive and negative concept evidence across frames, thereby providing clinically interpretable and traceable diagnostic support.

Appendix A.2 – Explainability in Medical Imaging

In parallel, MedCLIP (Wang et al., 2022) learns cross-modal representations from unpaired medical images and text through knowledge-guided semantic matching. MIChain instead aligns unpaired BI-RADS concept descriptions and ultrasound video features in a shared concept space, and further performs evidence-guided video reasoning to aggregate reliable concept evidence across frames for multimodal diagnosis. This design bridges unpaired medical vision-language learning and evidence-based reasoning for interpretable ultrasound video diagnosis.

Response to W2

Following the reviewer's suggestions, we have extended the baseline comparisons in Table 1 to include CVA-Net and additional recent methods, and introduced experiments on the public Breast-USV dataset for broader comparisons with domain-specific approaches.

1)Additional comparisons on BUS-VD. We include CVA-Net (MICCAI'22), CRL (MICCAI'25), and MVP-CBM (IJCAI'25) in Table 1. The corresponding results are summarized below.

Table 1. Additional baseline comparisons on BUS-VD (%).

MethodGrading AccGrading KappaGrading F1Classification AccClassification KappaClassification F1
CVA-Net (MICCAI'22)81.1168.6677.5582.4947.7073.37
CRL (MICCAI'25)79.7265.3260.1782.0351.6975.81
MVP-CBM (IJCAI'25)86.6473.1776.9788.0267.2283.56
MIChain (Ours)87.1077.4682.3888.9469.3784.62

As shown above, MIChain consistently outperforms these methods in both BI-RADS grading and benign/malignant classification, demonstrating its effectiveness compared with additional recent baselines.

2)Additional comparisons on the public Breast-USV dataset. To further evaluate MIChain against breast ultrasound-specific approaches, we introduce the public Breast-USV dataset (Yan et al., 2026) and conduct extensive comparisons with both general-purpose and breast ultrasound-specific methods. Representative results are summarized below, while the complete comparisons are reported in Table 2 of the revised manuscript.

Table 2. Representative benign/malignant classification results on Breast-USV (%).

MethodAUC ↑Acc ↑Recall ↑Prec ↑F1 ↑
Video Swin (CVPR'22)91.8086.6787.6286.1086.81
⋮⋮⋮⋮⋮⋮
DL-video (EJR'23)88.7484.5282.5485.9684.20
KGA-Net (MICCAI'23)91.1086.5186.5186.6086.52
DTFN (IJACSA'24)89.8584.7284.1285.2684.65
CKC-FB1 (MedIA'26)93.1990.2889.2991.2090.20
MIChain (Ours)94.3391.6795.2488.8991.95

MIChain achieves the highest AUC (94.33%), Acc (91.67%), Recall (95.24%), and F1 (91.95%), outperforming the best competing results by 1.14, 1.39, 5.95, and 1.75 percentage points, respectively. These results demonstrate the effectiveness of MIChain compared with breast ultrasound-specific methods on a public benchmark.

The additional experiments and corresponding analyses have been incorporated into Sections 3.1–3.2 and Tables 1–2 of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer gTR8 (2/3)

Response to W3

We clarify that MIChain's reported inference latency of 47.61 ms is measured per 32-frame test video, rather than per individual frame, covering the complete online inference pipeline on an NVIDIA A100 80GB GPU.

The additional computational cost compared with most interpretable baselines arises primarily from MIChain's more comprehensive diagnostic pipeline. While image-based concept models such as CBM and CEM obtain video-level predictions by averaging frame-level outputs, MIChain further performs concept-specific evidence learning and reliability-aware aggregation across frames, together with lesion localization and boundary generation to provide traceable visual evidence. These additional components contribute to the inference overhead while enabling more comprehensive and interpretable diagnostic outputs.

Importantly, compared with non-interpretable video-based methods, MIChain achieves competitive inference efficiency while providing substantially improved diagnostic performance and clinical interpretability. Its 47.61 ms latency covers the entire 32-frame video and supports practical video-level clinical assistance under the evaluated hardware setting.

Response to Minor weaknesses

1)We have simplified Fig. 2 in the revised manuscript to highlight the overall workflow and key components of MIChain.

2)Regarding the notation, we have clarified the notation before Eq. (3) in the main paper, explicitly defining NvN_v and NtN_t as the numbers of visual and textual observations, respectively.

Response to Q1

To clarify the baseline training protocols, all compared methods are initialized using the pretrained checkpoints and initialization strategies recommended by their respective official implementations, rather than being trained from scratch. All baselines are fully fine-tuned on BUS-VD for 10,000 iterations without LoRA, with hyperparameters independently tuned for each model.

Regarding evaluation, video-based models directly process the 32 sampled frames, whereas image-based interpretable models process each sampled frame individually. Their frame-level prediction scores are averaged to obtain video-level BI-RADS grading, benign/malignant classification, and concept prediction results.

Further details on baseline initialization, training strategies, and evaluation protocols are provided in Appendix H.2.

Response to Q2

We clarify that the benign/malignant ground-truth labels in Table 1 are based on pathological diagnoses and are annotated by physicians. Regarding the prediction mechanism, MIChain performs BI-RADS grading and benign/malignant classification as two separate prediction tasks, each with its own classification head. Specifically, the benign/malignant prediction is generated directly by an independent binary classification head rather than being derived from the predicted BI-RADS grades.

AuthorsOpenReview ↗

Response to Reviewer gTR8 (3/3)

Q3. Per-Grade BI-RADS Grading Performance under Class Imbalance

To address the concern regarding class imbalance, we have added a per-grade BI-RADS grading analysis in Appendix C.4 (Table 9) of the revised manuscript, reporting class-wise Precision and F1-score for all compared methods. Representative results are summarized below.

Table 9. Per-grade BI-RADS grading performance on BUS-VD (%).

MethodG2 Prec.G2 F1G3 Prec.G3 F1G4 Prec.G4 F1G5 Prec.G5 F1Avg. Prec.Avg. F1
Video Swin100.0096.3084.3489.7477.7863.64100.0057.1490.5376.70
CVA-Net100.0096.3081.2987.7070.0051.22100.0075.0087.8277.55
UniFormerV276.4783.8787.4288.8971.1165.9875.0066.6777.5076.35
VideoMAEv2100.0092.3187.1890.0770.8368.00100.0033.3389.5070.93
InternVideo283.3376.9282.1487.9072.9760.670.000.0059.6156.37
VideoMamba90.9180.0082.1487.9077.7863.64100.0057.1487.7172.17
DejaVid75.0080.0087.7490.3779.0771.58100.0075.0085.4579.24
CBM72.2281.2589.1989.8066.6766.020.000.0057.0259.27
CEM80.0082.7687.2588.1467.3565.3575.0066.6777.4075.73
ProbCBM90.0075.0081.2987.7072.2259.090.000.0060.8855.45
ECBM73.6884.8588.1689.9376.9265.9357.1466.6773.9876.85
evi-CEM76.4783.8793.4890.8570.1873.3960.0060.0075.0377.03
SSCBM73.6884.8589.0489.0473.9169.3966.6772.7375.8379.00
CRL93.3396.5585.5387.2558.0056.860.000.0059.2160.17
MVP-CBM84.6281.4890.4890.7876.3678.50100.0057.1487.8676.97
MIChain (Ours)92.3188.8987.4291.1583.3374.47100.0075.0090.7782.38

As shown in Table 9, MIChain achieves the highest macro-averaged Precision (90.77%) and F1-score (82.38%). At the class level, it achieves the best F1-score for Grade 3 (91.15%) and the highest Precision for Grade 4 (83.33%). For the underrepresented Grade 5, MIChain achieves an F1-score of 75.00%, matching the best competing result. These results provide a more detailed assessment of MIChain's grading performance under class imbalance.

Reviewer_tahtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MIChain (Multimodal Interpretability Chain), a multimodal interpretable reasoning framework for BI-RADS classification diagnosis using breast ultrasound videos. The framework is intended to address several limitations of existing models, including reliance on manually selected frames, the requirement of paired diagnostic reports, black-box outputs, and the lack of traceable multimodal diagnostic evidence. Comprehensive experiments and theoretical validations are conducted on BUS-VD, a large-scale breast ultrasound video dataset constructed in this work.

Critical strengths

Under the linear Gaussian factor model and by leveraging Fisher information, this paper demonstrates that unpaired domain-specific clinical texts can still improve visual concept representations. The BUS-VD breast ultrasound video dataset is constructed, and its annotations consist of lesion bounding boxes, masks, 14 types of BI-RADS multi-labels, and grading labels. The training, validation, and test sets are partitioned at the patient level to prevent frame leakage from the same patient across splits. Furthermore, dual-aware evidence learning coupled with temporal evidence aggregation is proposed. Using evidence sufficiency and directional certainty, valid ultrasound frames are selected automatically.

Critical weaknesses

(1) The theorems rely on strong linear Gaussian assumptions, whereas deep networks are nonlinear and non-Gaussian; experimental verification of the theory is claimed, but direct measurement of concept-representation estimation variance is absent, and attribution of gains to Fisher information rather than regularization or class priors remains unsupported. (2) ``Unpaired'' is defined as non-one-to-one video--text matching under a shared BI-RADS ontology, but no control group with irrelevant-domain texts is included; whether only BI-RADS-relevant texts improve performance is therefore unverified. (3) Although an inference latency of 47.61 ms is reported for the model integrating DINOv2, BERT, Grounding-DINO, PiDiNet, and temporal evidence aggregation, GPU memory consumption, total parameter count, and per-submodule time breakdowns are not provided. (4) Temporal evidence aggregation uses only weighted summation of reliability scores; each frame is treated as an independent sample, so spatial adjacency and geometric transformation relationships between consecutive slices are not modeled, and temporal information is not exploited. (5) Subjective logic with the Beta distribution is used for evidence uncertainty, but calibration experiments such as ECE and reliability diagrams are not conducted; the uncertainty scores may therefore not reflect the actual risk of misdiagnosis.

Questions for authors

(1) The theorems rely on strong linear Gaussian assumptions, whereas deep networks are nonlinear and non-Gaussian; experimental verification of the theory is claimed, but direct measurement of concept-representation estimation variance is absent, and attribution of gains to Fisher information rather than regularization or class priors remains unsupported. (2) ``Unpaired'' is defined as non-one-to-one video--text matching under a shared BI-RADS ontology, but no control group with irrelevant-domain texts is included; whether only BI-RADS-relevant texts improve performance is therefore unverified. (3) Although an inference latency of 47.61 ms is reported for the model integrating DINOv2, BERT, Grounding-DINO, PiDiNet, and temporal evidence aggregation, GPU memory consumption, total parameter count, and per-submodule time breakdowns are not provided. (4) Temporal evidence aggregation uses only weighted summation of reliability scores; each frame is treated as an independent sample, so spatial adjacency and geometric transformation relationships between consecutive slices are not modeled, and temporal information is not exploited. (5) Subjective logic with the Beta distribution is used for evidence uncertainty, but calibration experiments such as ECE and reliability diagrams are not conducted; the uncertainty scores may therefore not reflect the actual risk of misdiagnosis. (6) The compared baseline models (e.g., Video Swin, VideoMAEv2, CBM, and CEM) are mostly used in their original versions. However, it is not specified whether identical preprocessing, input resolution, and hyperparameter tuning strategies are used for these baselines, nor whether additional prediction heads are appended to adapt them to the BI-RADS task.

4 comments
AuthorsOpenReview ↗

Response to Reviewer taht (1/3)

We appreciate the reviewer's positive assessment of our work and the constructive feedback, which has helped us further clarify and strengthen the manuscript.

Response to Q1&W1

Our theoretical analysis aims to characterize the information complementarity between unpaired modalities through a tractable linear-Gaussian shared-factor model, rather than to approximate the entire nonlinear deep network. As formulated in Eqs. (2)–(4), visual and textual observations share the underlying clinical concept factors θc\theta_c. Under conditional independence, their Fisher information is additive, i.e., Jv,t=Jv+JtJ_{v,t}=J_v+J_t. Accordingly, Theorem 1 establishes reduced estimation covariance under its identifiability assumptions, while Theorem 2 characterizes information gain along concept directions covered by textual observations.

To connect this theoretical insight to MIChain, we further clarify how the complementary information is exploited in the actual network. As described in Section 2.1, visual and textual embeddings share the same projection parameters WW and bb. Consequently, their supervision signals jointly contribute to optimizing the shared projector. Specifically, the gradient with respect to WW decomposes as:

∇WL=∑t∈Vδtvis(htvis)⊤+∑j∈Tδjtxt(hjtxt)⊤\nabla_W \mathcal{L} = \sum_{t\in\mathcal{V}} \delta_t^{\mathrm{vis}}(h_t^{\mathrm{vis}})^\top + \sum_{j\in\mathcal{T}} \delta_j^{\mathrm{txt}}(h_j^{\mathrm{txt}})^\top

This demonstrates how unpaired textual supervision contributes additional optimization signals to the shared projector without requiring sample-wise video–text correspondence, as further derived in Appendix F of the revised manuscript.

Response to Q2&W2

To verify whether the performance gains arise from BI-RADS-specific textual semantics, we conduct additional controlled experiments by replacing the original BI-RADS descriptions with semantically meaningless text and medical text unrelated to breast lesions. These experiments are evaluated under both the shared-projector-only and full MIChain configurations, with results presented in Table 8.

Table 8. Ablation of textual content under different configurations (%).

Text ConditionUnpaired TextShared Proj.Evidence Reason.Evidence Aggr.Grading (Acc.)Grading (Kappa)Concept Prediction (AUPR)Concept Prediction (F1)
Shared Projector Only
No text×✓××83.4168.7358.9854.86
Meaningless text✓✓××82.9566.9458.2354.44
Irrelevant medical text✓✓××82.0366.9757.1354.02
BI-RADS text✓✓××84.7972.2259.3255.23
Full MIChain
No text×✓✓✓86.1875.1259.5658.96
Meaningless text✓✓✓✓85.7172.9659.2256.68
Irrelevant medical text✓✓✓✓83.8771.8758.1458.07
BI-RADS text✓✓✓✓87.1077.4660.9260.92

The results show that BI-RADS descriptions consistently achieve the best performance across all four metrics under both configurations, while meaningless and irrelevant medical texts perform below the corresponding no-text baselines. These findings demonstrate the importance of BI-RADS-specific clinical semantics for shared concept representation learning and diagnostic performance. The additional experiments are included in Appendix C.2 (Table 8), with a corresponding discussion added to Section 3.3 of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer taht (2/3)

Response to Q3&W3

We report the model parameters, peak GPU memory consumption, and per-submodule inference latency, as summarized below. All measurements are conducted on an NVIDIA A100 80GB GPU.

Table. Computational complexity and inference efficiency of MIChain.

MetricComponentResult
Model ParametersOnline resident parameters85.899 M
Parameters on measured online path85.898 M
Offline BERT encoder109.482 M
GPU MemoryPeak allocated (online inference)0.514 GB
Inference LatencyDINOv243.734 ms
Shared Projector0.872 ms
Dual-Aware Evidence Learning + Video Evidence Aggregation1.502 ms
Grounding-DINO44.750 ms
PiDiNet42.870 ms
Total Online Inference47.610 ms
Offline ProcessingBERT Text Feature Extraction (14 concepts)60.929 ms

Grounding-DINO and PiDiNet operate in parallel with the diagnostic branch; therefore, their individual inference times overlap and are not additive. The total online inference latency is 47.61 ms per video. In addition, BERT text features are precomputed offline and do not introduce additional per-video inference overhead. These computational details are included in Appendix H.1 of the revised manuscript.

Response to Q4&W4

We acknowledge that the current framework does not explicitly model spatial adjacency or geometric transformations between consecutive frames. This design is motivated by the clinical characteristics of breast ultrasound examinations. During routine examinations, radiologists manually manipulate the probe to scan lesions from different angles and imaging planes. Consequently, the scanning trajectory and inter-frame geometric relationships may vary across examinations depending on the operator's scanning technique. As described in our Introduction, ultrasound videos capture lesion morphology from multiple probe angles and imaging planes, and the same lesion may exhibit different characteristics across different imaging planes. Therefore, MIChain focuses on integrating diagnostic evidence across multiple views rather than explicitly reconstructing inter-frame spatial relationships.

Specifically, Eq. (8) extracts frame-level positive and negative concept evidence and evidential uncertainty Ut,cU_{t,c}. Eq. (9) evaluates evidence reliability by jointly considering evidential sufficiency and directional certainty, yielding the following reliability score:

Rt,c=∣et,c+−et,c−∣et,c++et,c−+2R_{t,c}=\frac{\lvert e_{t,c}^{+}-e_{t,c}^{-}\rvert}{e_{t,c}^{+}+e_{t,c}^{-}+2}

Based on this reliability, Eq. (10) assigns concept-specific frame weights:

wt,c=exp⁡(Rt,c/η)∑t′=1Texp⁡(Rt′,c/η)w_{t,c}=\frac{\exp(R_{t,c}/\eta)}{\sum_{t'=1}^{T}\exp(R_{t',c}/\eta)}

The corresponding frame-level evidence is then aggregated as:

vc=∑t=1Twt,cet,c\mathbf{v}_c=\sum_{t=1}^{T}w_{t,c}\mathbf{e}_{t,c}

As shown in the ablation study in Table 5, evidence aggregation consistently improves diagnostic grading and concept prediction performance, as summarized below.

Table 5. Ablation studies of the key components in ourmodel.

Evidence AggregationGrading AccGrading KappaConcept AUPRConcept F1
Without84.3372.4556.4159.24
With87.1077.4660.9260.92
Improvement+2.77+5.01+4.51+1.68

These results demonstrate that reliability-guided evidence aggregation improves both diagnostic grading and concept prediction by integrating informative evidence across ultrasound frames.

AuthorsOpenReview ↗

Response to Reviewer taht (3/3)

Response to Q5&W5

To assess the reliability of MIChain's predictive probabilities, we conduct additional calibration experiments using Expected Calibration Error (ECE) for both concept prediction and BI-RADS grading. The results are summarized below.

TaskECE (%) ↓
Concept Prediction6.68
BI-RADS Grading7.82

The results show ECE values of 6.68% for concept prediction and 7.82% for BI-RADS grading, indicating relatively small discrepancies between predicted confidence and empirical accuracy. These findings provide quantitative evidence of probability calibration in MIChain.

Response to W6

To ensure fair and reproducible comparisons, we clarify the preprocessing procedures, hyperparameter tuning strategies, and task-specific adaptations applied to all baseline methods.

1)Regarding data preprocessing, all baselines follow the same preprocessing pipeline as MIChain, with 16 frames sampled during training and 32 during validation at an interval of 5, resized and cropped to 224 × 224. For image-based interpretable methods, each sampled frame is processed independently, and the frame-level prediction scores are averaged to obtain video-level predictions for BI-RADS grading, benign/malignant classification, and concept prediction.

2)Regarding hyperparameter tuning, we adopt the pretrained weights and initialization strategies recommended by the official implementations. Rather than enforcing identical hyperparameter values across different architectures, we independently tune each baseline to accommodate its architectural characteristics.

3)Regarding task adaptation, we append task-specific prediction heads to the original baseline networks for BI-RADS grading, while keeping all other architectural components unchanged from their official implementations.

These implementation details are provided in Appendix H.2 of the revised manuscript.

AuthorsOpenReview ↗

Follow-up on Our Revisions and Invitation for Further Discussion

Thank you for your careful review and constructive suggestions, which have helped us further strengthen the manuscript.

In response to your comments, we have conducted additional experiments and made several revisions, summarized as follows:

  • W1&Q1 The theoretical analysis has been further clarified to characterize information complementarity between unpaired modalities under the linear-Gaussian shared-factor model. Additional derivations explain how unpaired textual supervision contributes to optimizing shared concept representations (Section 2.2, Appendices E–F).

  • W2&Q2 Additional ablation experiments compare BI-RADS descriptions with meaningless and breast-lesion-irrelevant medical texts under both shared-projector-only and full MIChain configurations. The results demonstrate that BI-RADS descriptions consistently achieve the best performance, while both control conditions perform worse than the no-text baselines (Section 3.3, Appendix C.2, Table 8).

  • W3&Q3 The computational complexity and inference efficiency analyses have been expanded to include model parameters, peak GPU memory consumption, and per-submodule inference latency. The results further clarify the computational cost of MIChain's complete diagnostic pipeline (Appendix H.1, Table 11).

  • W4&Q4 The clinical motivation for reliability-guided evidence aggregation across ultrasound frames has been further clarified, emphasizing the integration of diagnostic information from different imaging planes. The ablation results demonstrate the effectiveness of this strategy in improving grading and concept prediction performance (Sections 2.3 and 3.3, Table 5).

  • W5&Q5 Additional probability calibration experiments using Expected Calibration Error (ECE) have been conducted for both concept prediction and BI-RADS grading, providing quantitative assessments of predictive confidence (Our detailed response).

  • W6 The baseline evaluation protocols have been further clarified, including consistent data preprocessing, model-specific hyperparameter tuning, pretrained initialization, and task-specific prediction heads, to improve the fairness and reproducibility of experimental comparisons (Appendix H.2).

Your thoughtful suggestions have helped us strengthen the theoretical explanations, experimental evaluation, and reproducibility of our work. We sincerely appreciate your contributions to improving the quality of our manuscript.

We would be grateful to know whether these revisions have adequately addressed your concerns. If any questions or concerns remain, we would welcome your further comments and would be happy to provide additional clarification or make further improvements accordingly.

Thank you again for your time and careful consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Thank you to all reviewers and AC.

We sincerely appreciate the time and effort invested by the reviewers in evaluating our manuscript, as well as their constructive and encouraging feedback. We are grateful for their recognition of our work’s contributions and the thoughtful comments that have helped us improve the paper.

Below, we summarize the key strengths highlighted by the reviewers and the main revisions made in response to their valuable feedback:

Strengths:

  • The paper proposes a novel framework for interpretable breast ultrasound video diagnosis that integrates unpaired clinical knowledge, dual-aware evidence learning, and reliability-guided video evidence aggregation to construct traceable multimodal diagnostic chains. The framework enables automatic identification of diagnostic evidence from ultrasound videos without relying on manually selected frames or paired diagnostic reports. (Reviewers taht, gTR8, pvqp)

  • The paper provides theoretical and empirical support for learning from unpaired clinical knowledge, combining Fisher information analysis with comprehensive experimental evaluations. The theoretical formulation explains how unpaired clinical text can contribute complementary information to shared concept representation learning, supporting the methodological design of MIChain. (Reviewers taht, gTR8)

  • The paper addresses the clinically significant challenge of explainability in breast ultrasound video diagnosis by automatically identifying informative diagnostic evidence and providing both visual and textual explanations. By integrating lesion localization, clinical concepts, and diagnostic predictions into an interpretable chain, the framework brings AI-assisted diagnosis closer to radiologists' clinical workflow and demonstrates potential for clinical applications. (Reviewers gTR8, pvqp)

Key Revisions and Additional Experiments:

  • To address concerns regarding cross-modal semantic alignment and the effectiveness of unpaired clinical concept text, this paper further clarifies the shared concept representation learning mechanism (Section 2.2, Appendices F). More importantly, additional ablation experiments compare BI-RADS descriptions with meaningless text and breast-lesion-irrelevant medical text under both shared-projector-only and full MIChain configurations. The results demonstrate that BI-RADS descriptions consistently achieve the best performance, while both control conditions perform worse than the no-text baselines (Section 3.3, Appendix C.2, Table 8).(Reviewers taht, pvqp)

  • In response to the reviewers' suggestions on external generalization and the comprehensiveness of baseline comparisons, this paper introduces additional experiments on the public Breast-USV dataset, which was introduced in a concurrent study, demonstrating MIChain's superior diagnostic performance and generalization capability (Sections 3.1–3.2, Table 2). Furthermore, the baseline comparisons are expanded to include recent methods, particularly those proposed in 2025 (Sections 3.1–3.2, Table 1). (Reviewers gTR8, pvqp)

  • Regarding video evidence aggregation and the faithfulness of diagnostic interpretability, this paper further clarifies the clinical motivation for reliability-guided evidence aggregation across ultrasound frames (Section 2.3). The ablation studies demonstrate the effectiveness of evidence-guided video reasoning (Section 3.3, Table 5), while the concept intervention experiments verify that modifying clinical concept scores directly influences the final diagnostic predictions (Section 3.2, Fig. 4). (Reviewers taht, pvqp)

  • To further improve experimental reliability and reproducibility, this paper supplements class-wise BI-RADS grading analyses (Appendix C.3, Table 9), clarifies dataset annotations and baseline evaluation protocols (Appendices B and H.2), and provides additional computational complexity and inference efficiency analyses (Appendix H.1). Furthermore, the related work discussion is expanded to better distinguish MIChain from existing approaches (Appendix A). (Reviewers taht, gTR8, pvqp)

Based on the reviewers’ valuable suggestions and comments, we have conducted additional experiments and enhanced the clarity of the manuscript as requested. We believe these improvements have significantly strengthened the quality of our work.

Thank you again for your professional and careful review comments. Your encouragement and suggestions motivate us to continue contributing to the community.

If you have any further questions or concerns, we would be happy to address them. Your feedback will be highly appreciated.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.