- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a multimodal evidence reasoning framework for interpretable breast ultrasound video diagnosis using unpaired clinical text and video evidence. The framework identifies key diagnostic frames, combines visual and textual evidence, and generates explainable diagnostic chains to support predictions. Extensive experiments on a large-scale breast ultrasound video dataset demonstrate its effectiveness in diagnostic performance and interpretability.
Critical strengths
This paper addresses the issue of explainability in breast ultrasound video diagnosis. It proposes a diagnostic framework that combines unpaired clinical text with reasoning based on video evidence. The proposed method can automatically extract key diagnostic information and provide both visual and textual explanations. It also shows potential for clinical use.
Critical weaknesses
- The frame-level weighted aggregation in Equation (10) is insensitive to the order of the input frames. It does not model temporal order or changes between frames. Therefore, the contribution of temporal information in continuous videos remains unclear.
- Shared projection parameters and improved downstream performance are not enough to demonstrate cross-modal semantic alignment. The proposed text-based semantic guidance without paired data lacks direct validation.
- The authors have shown that concept scores contribute to predictions, the roles of detection boxes, boundary maps, and text descriptions in the final predictions remain unclear. The current results do not fully demonstrate that the complete diagnostic process faithfully reflects the model's decision-making.
- The baselines include recent general video models, they cover only a limited range of methods.
- The paper describes a case-level data split but does not clarify whether each case corresponds to a unique patient. It also lacks key details about the data sources. Therefore, patient-level independence and experimental reproducibility remain difficult to assess.
- The experiments are conducted on only one private dataset, without independent external validation. The current results are not sufficient to support the model's ability to generalize across different data sources.
- The reference standard for benign and malignant labels is unclear. Agreement among doctors on these labels is also not reported.
Questions for authors
- The authors should explain what temporal information in continuous videos contributes compared with a set of independent ultrasound images. According to Equation (10), if each frame is encoded independently and then combined through weighted summation, simply changing the order of the input frames will not affect the final result.
- The authors should explain whether unpaired text truly achieves cross-modal semantic alignment. Sharing projection parameters does not necessarily mean that the two modalities are semantically aligned (Section 2.1).
- The authors should further clarify whether the proposed Multimodal Explainable Diagnostic Chain directly affects the final prediction or only provides an explanation after the prediction. For example, if key visual evidence is replaced or removed, or if concept descriptions and their corresponding scores are modified, will the final prediction change in a clinically meaningful way?
- The authors should include comparisons with more baseline methods proposed in recent years.
- The authors should clarify whether the training, validation, and test sets of the private dataset are independent at the patient level. More details about the data sources should also be provided. In addition, the authors should state whether the study received approval from the relevant ethics committee.
- The authors should report results on external datasets, such as [1] [2], to evaluate the generalization ability of the proposed method. Since Table 3 already compares the method proposed in [2], the authors should also explain why the dataset from [2] was not included in the external evaluation.
- The authors should explain how the labels were obtained. Specifically, they should clarify whether the labels were based on pathology results, clinical follow-up, or BI-RADS category mapping. An analysis of agreement among doctors on these labels should also be provided.
Ref [1] Clinical Knowledge Constrained Multi-Task Learning Framework for Breast Cancer Diagnosis Using Ultrasound Videos [2] A New Dataset and a Baseline Model for Breast Lesion Detection in Ultrasound Videos
6 comments
Response to Reviewer pvqp (2/4)
Response to Q2&W2
We clarify that MIChain promotes cross-modal semantic alignment by mapping visual and textual embeddings into a shared BI-RADS concept space, where both modalities jointly contribute to learning clinically meaningful concept representations through shared projection parameters and concept-level supervision.
1)Regarding the alignment mechanism, as described in Section 2.1, visual and textual embeddings are mapped into a shared concept space using the same learnable projection parameters:
Importantly, the semantic correspondence is defined by the shared BI-RADS concept ontology: both modalities are represented over the same set of clinical concept slots, each associated with a corresponding BI-RADS description. Furthermore, visual concept supervision and textual supervision through provide learning signals for these shared representations. Thus, the alignment is established at the clinical concept level, without requiring sample-wise video–text correspondence.
2)Regarding the theoretical basis, Eqs. (2)–(3) formulate visual and textual observations as independent observations of the same shared clinical concept factors . Their Fisher information is therefore additive (Eq. (4)), i.e.,
Theorems 1 and 2 further establish increased information and reduced estimation uncertainty under the stated assumptions (Eqs. (5)–(7)), providing a theoretical rationale for incorporating unpaired clinical text into shared concept representation learning.
3)To further validate the contribution of BI-RADS-specific semantics, we conduct additional controlled experiments using semantically meaningless text and medical text unrelated to breast lesions. We evaluate these alternatives under both the shared-projector-only and full evidence-guided video reasoning configurations. The results are summarized in Table 8.
Table 8. Ablation of textual content under different configurations (%).
| Text Condition | Unpaired Text | Shared Proj. | Evidence Reason. | Evidence Aggr. | Grading (Acc.) | Grading (Kappa) | Concept Prediction (AUPR) | Concept Prediction (F1) |
|---|---|---|---|---|---|---|---|---|
| Shared Projector Only | ||||||||
| No text | × | ✓ | × | × | 83.41 | 68.73 | 58.98 | 54.86 |
| Meaningless text | ✓ | ✓ | × | × | 82.95 | 66.94 | 58.23 | 54.44 |
| Irrelevant medical text | ✓ | ✓ | × | × | 82.03 | 66.97 | 57.13 | 54.02 |
| BI-RADS text | ✓ | ✓ | × | × | 84.79 | 72.22 | 59.32 | 55.23 |
| Full MIChain | ||||||||
| No text | × | ✓ | ✓ | ✓ | 86.18 | 75.12 | 59.56 | 58.96 |
| Meaningless text | ✓ | ✓ | ✓ | ✓ | 85.71 | 72.96 | 59.22 | 56.68 |
| Irrelevant medical text | ✓ | ✓ | ✓ | ✓ | 83.87 | 71.87 | 58.14 | 58.07 |
| BI-RADS text | ✓ | ✓ | ✓ | ✓ | 87.10 | 77.46 | 60.92 | 60.92 |
As shown in Table 8, BI-RADS descriptions consistently outperform both control text conditions across all four metrics under both configurations. Moreover, both control conditions perform below their corresponding no-text baselines. These results support the importance of BI-RADS-specific clinical semantics for shared concept representation learning, rather than merely introducing additional textual inputs. The complete experiments are provided in Appendix C.2 (Table 8), with a corresponding discussion in Section 3.3 of the revised manuscript.
Response to Reviewer pvqp (3/4)
Response to Q3&W3
In MIChain, the final BI-RADS prediction is directly derived from the diagnostic evidence processed through the proposed Multimodal Explainable Diagnostic Chain, rather than being independently generated before the explanations. This is further demonstrated by the ablation studies of Evidence-Guided Video Reasoning and the concept intervention experiments.
1)Regarding the contribution of visual evidence to the final prediction, MIChain introduces Evidence-Guided Video Reasoning to extract and aggregate frame-level diagnostic evidence. Specifically, Dual-Aware Evidence Learning assesses the reliability of concept evidence within each frame, while Video Evidence Aggregation assigns concept-specific weights to emphasize frames providing more reliable evidence for different clinical concepts and suppress the influence of ambiguous or low-quality frames.
The ablation results in Table 5 demonstrate the contribution of these components:
| Configuration | Evidence Learning | Evidence Aggregation | Acc. | Kappa | AUPR | F1 |
|---|---|---|---|---|---|---|
| Without reasoning | × | × | 84.79 | 72.22 | 59.32 | 55.23 |
| Evidence learning only | ✓ | × | 84.33 | 72.45 | 56.41 | 59.24 |
| Complete reasoning | ✓ | ✓ | 87.10 | 77.46 | 60.92 | 60.92 |
These results demonstrate that identifying and prioritizing key visual evidence directly contributes to concept prediction and final grading, rather than merely providing post-hoc explanations.
2)Regarding whether modifying clinical concept scores can affect the final prediction, MIChain employs 14 clinical concepts derived from BI-RADS sonographic descriptors. Each predicted concept probability is associated with its corresponding clinical description to provide interpretable textual evidence. More importantly, directly gates the positive and negative evidence to construct the evidential concept bottleneck (Eq. (12)):
Following the test-time concept intervention paradigm of Concept Bottleneck Models (CBMs) [1], MIChain allows the predicted scores of clinical concepts to be modified and propagated to the final prediction. Specifically, intervention is performed by replacing the original concept probability with a corrected value :
The modified evidential concept bottleneck is then passed to the grading head:
The existing intervention results in Fig. 4(b) demonstrate the impact of modifying concept scores on the final diagnostic performance:
| Task | Original Scores | Corrected Scores | Improvement |
|---|---|---|---|
| Grading Accuracy | 87.1% | 89.4% | +2.4 pp |
| Classification Accuracy | 88.9% | 89.8% | +0.9 pp |
Furthermore, Fig. 4(a) shows that MIChain consistently improves as the proportion of corrected concept scores increases, outperforming CBM and CEM across different intervention degrees. These results demonstrate that modifying the predicted scores of clinically meaningful concepts directly influences the final diagnostic prediction.
Reference
[1] Koh et al. Concept Bottleneck Models. International Conference on Machine Learning (ICML), 2020.
Response to Q4&W4
Following the reviewer's suggestion, we have expanded the baseline comparisons in Table 1 of the revised manuscript to include additional methods, particularly those proposed in 2025. Specifically, we have incorporated CVA-Net (MICCAI'22), CRL (MICCAI'25), and MVP-CBM (IJCAI'25). The corresponding results are summarized below.
Table 1. Additional baseline comparisons on BUS-VD (%).
| Method | Grading Acc | Grading Kappa | Grading F1 | Classification Acc | Classification Kappa | Classification F1 |
|---|---|---|---|---|---|---|
| CVA-Net (MICCAI'22) | 81.11 | 68.66 | 77.55 | 82.49 | 47.70 | 73.37 |
| CRL (MICCAI'25) | 79.72 | 65.32 | 60.17 | 82.03 | 51.69 | 75.81 |
| MVP-CBM (IJCAI'25) | 86.64 | 73.17 | 76.97 | 88.02 | 67.22 | 83.56 |
| MIChain (Ours) | 87.10 | 77.46 | 82.38 | 88.94 | 69.37 | 84.62 |
As shown above, MIChain consistently outperforms these additional baselines across all six reported metrics for BI-RADS grading and benign/malignant classification. These results further demonstrate the effectiveness of MIChain compared with recent methods.
Response to Reviewer pvqp (1/4)
We sincerely appreciate your recognition of the clinical motivation and interpretability of our framework. Thank you for your constructive comments regarding video reasoning, cross-modal semantic alignment, diagnostic faithfulness, and dataset validation. We respond to your concerns below.
Response to Q1&W1
We acknowledge that Eq. (10) is permutation-invariant and does not explicitly model inter-frame temporal transitions. This design is motivated by the clinical characteristics of breast ultrasound examinations. During routine examinations, radiologists manually manipulate the probe to scan lesions from different angles and imaging planes. Consequently, the chronological order of frames is largely determined by the operator's scanning trajectory and technique, which may vary across examinations. As described in our Introduction, ultrasound videos capture lesion morphology from multiple probe angles and imaging planes, providing more comprehensive diagnostic information than individual images. Moreover, the same lesion may exhibit different characteristics across different imaging planes.
Accordingly, MIChain focuses on aggregating reliable diagnostic evidence across ultrasound frames rather than explicitly modeling acquisition order. Specifically, Eq. (8) extracts frame-level positive and negative evidence, together with evidential uncertainty Ut,c. Eq. (9) further evaluates frame-level reliability by jointly considering evidential sufficiency and directional certainty, yielding the following reliability score:
Rt,c=et,c++et,c−+2∣et,c+−et,c−∣
Based on this reliability, Eq. (10) computes concept-specific frame weights:
wt,c=∑t′=1Texp(Rt′,c/η)exp(Rt,c/η)
The corresponding frame-level evidence is then aggregated as:
vc=∑t=1Twt,cet,c
The ablation results in Table 5 demonstrate the effectiveness of evidence aggregation:
Table 5. Ablation studies of the key components in ourmodel.
Thus, MIChain adaptively aggregates diagnostic evidence across ultrasound frames according to its estimated reliability for each concept. As shown in Table 5, introducing evidence aggregation improves grading accuracy from 84.33% to 87.10% and Kappa from 72.45% to 77.46%, supporting the effectiveness of this design.