- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SuG uses anatomy and lesion segmentation labels together with medical reports to train a model for CT diagnosis and lesion localization. It combines anatomy features at different scales and uses lesion labels to guide attention. The experiments cover chest and abdominal diseases, lesion localization, and performance under different training and testing conditions.
Critical strengths
- The main idea is well motivated. Anatomy labels and lesion labels give the model useful information at different levels. The full model improves lesion localization over the specialist-only variant. It also performs better on an external kidney dataset without using kidney tumor or cyst segmentation labels during training. Together with the diagnosis results, this supports the value of combining specialist knowledge in a broader model.
- The evaluation covers many practical settings. The authors test joint chest and abdomen training, reduced segmentation labels, changes to report parsing, corrupted reports, and report generation. These tests give useful evidence that the method works under different conditions, even though they do not explain every interaction between its parts.
Critical weaknesses
- Some reported results are inconsistent. The pancreatic and gastric cancer results differ between the lesion diagnosis tables and the detailed disease results in the appendix, even though the category mapping appears to match. The authors should explain whether the prompts, test cases, or model checkpoints differ. The liver category also mixes cysts with different types of tumors, so its match to the specialist's training labels needs clarification. These issues make some comparisons harder to interpret, but they do not by themselves invalidate the broader diagnosis and localization results.
- Some claims go beyond the comparisons shown. The statistical tests use VLWS even where OpenVocabCT or nnUNet-fuse reports stronger results in the relevant comparison. Those tests therefore do not establish an improvement over the strongest alternatives. Also, showing that attention overlaps a lesion supports localization, but does not show that the model's diagnostic reasoning is reliable.
Questions for authors
- Why do the pancreatic and gastric cancer results differ across the lesion diagnosis and detailed disease tables? Does the liver specialist use the same disease definitions as the combined liver category?
- Could you compare against the strongest relevant baselines in the statistical tests, especially nnUNet-fuse for the abdominal lesion comparison? Otherwise, please limit the claims to the comparisons that were actually tested.
3 comments
Response to Reviewer dejC (Part 2 of 3)
2. Concerns regarding statistical significance analysis (W2, Q2)
Thank you for pointing this out. We have now included direct statistical comparisons against the strongest relevant baselines, particularly nnUNet-fuse for the abdominal lesion diagnosis task, as highlighted by the reviewer. We have carefully re-examined the baseline selection in Tables A18–A24 and recalculated the corresponding statistical significance results.
Baseline selection criteria and selected comparators. In the revised analysis, we select the strongest non-SuG baseline based on the highest average performance under each dataset and evaluation setting, and consistently compare all metrics within each table against this baseline. Specifically, the selected baselines are VLWS for A18, HCFNet for A19, OpenVocabCT for A20, HCFNet for A21 and A22, OpenVocabCT for A23, and nnUNet-fuse for A24. Among these, the baseline selections for Tables A20, A23, and A24 differ from those in the original manuscript. The revised statistical significance results for these three tables are presented below.
Results. These revised results demonstrate that SuG continues to achieve statistically significant improvements on the majority of metrics when compared against the strongest baselines under the corresponding evaluation settings. In particular, on LesionSegAbdomen, SuG achieves statistically significant improvements over nnUNet-fuse in average AUC and average specificity, while the difference in average sensitivity does not reach statistical significance. We have updated the corresponding statistical significance results in revised Appendix Tables A20, A23 and A24 to reflect these comparisons.
| Metric | OpenVocabCT | SuG (Improvement) | p-value |
|---|---|---|---|
| AUC | 83.2 | 85.1 (+1.89) | *** |
| F1 | 59.3 | 60.7 (+1.42) | ** |
Table A20 (updated): Bootstrap significance analysis on Merlin for unbalanced metrics.
| Metric | OpenVocabCT | SuG (Improvement) | p-value |
|---|---|---|---|
| AUC | 89.7 | 93.7 (+3.99) | *** |
| Sensitivity | 83.4 | 86.8 (+3.38) | * |
| Specificity | 84.2 | 88.0 (+3.74) | *** |
Table A23 (updated): Bootstrap significance analysis on MedVL-CT69K for the four tumor categories.
| Metric | nnUNet-fuse | SuG (Improvement) | p-value |
|---|---|---|---|
| AUC | 94.7 | 96.1 (+1.41) | * |
| Sensitivity | 89.2 | 89.4 (+0.20) | |
| Specificity | 89.0 | 91.7 (+2.71) | *** |
Table A24 (updated): Bootstrap significance analysis on LesionSegAbdomen.
Response to Reviewer dejC (Part 3 of 3)
3. Concern regarding the interpretation of lesion grounding (W2)
We thank the reviewer for this important clarification. We agree that spatial overlap between attention maps and lesion masks, while providing evidence of lesion localization, does not by itself establish causal faithfulness or the reliability of the model’s diagnostic reasoning.
Our grounding evaluation is intended to assess whether text-conditioned attention maps are spatially aligned with lesion annotations. The lesion-guided calibration loss explicitly supervises these maps using lesion masks, and the consistent improvements in AUC, AUPR, and Cohen’s d in Table 6 demonstrate improved lesion-focused localization and spatial interpretability. These grounding results are complementary to the diagnostic classification results, rather than evidence that the final diagnostic decision is causally determined by the attended lesion region.
For clarity, we have revised potentially overstrong wording in the manuscript. For example, we have replaced “trustworthy/reliable/faithful grounding” and “diagnostic traceability” with “improved lesion localization” and “spatial interpretability.” Establishing the causal faithfulness of localized evidence to diagnostic decisions is an important direction for future work.
Response to Reviewer dejC (Part 1 of 3)
We appreciate your recognition of our motivation and comprehensive evaluation. We also thank the reviewer for the valuable suggestions, which have helped improve the quality of our manuscript. Here we address your concerns point by point below.
1. Concerns regarding inconsistent results in appendix (W1, Q1)
Thank you for pointing this out. The four-category lesion evaluation in Tables 5/A26 and the per-disease evaluation in Table A27 follow different task definitions: the former focuses on organ-specific tumor categories, whereas the latter evaluates individual diseases. For the corresponding pairs of "pancreatic tumor" and "pancreatic cancer", as well as "gastric tumor" and "gastric cancer", on MedVL-CT69K, we use the same SuG model checkpoint, test cases, and ground-truth labels. The apparent differences in the reported results are attributable to the different task-specific text prompts.
Task definitions and evaluation consistency.The categories in the four-category lesion diagnosis task can encompass multiple pathological subtypes rather than being restricted to a single cancer type. For example, the pancreatic tumor annotations used for specialist pretraining also cover pancreatic neuroendocrine tumors (PNETs) and solid pseudopapillary tumors (SPTs). Therefore, the "pancreatic tumor" category is broader than the specific diagnosis of "pancreatic cancer". We maintain a consistent prompting strategy for the four-category lesion diagnosis task across LesionSegAbdomen and MedVL-CT69K. Although the pancreatic and gastric tumor categories in MedVL-CT69K happen to correspond exclusively to pancreatic cancer and gastric cancer, respectively, we do not modify the task-specific prompts accordingly. All competing vision–language methods follow the same evaluation protocol within each task to ensure fair comparisons.Task-specific prompts.The specific positive and negative prompts are provided below (Table R5). The prompts used in Table A27 additionally incorporate imaging descriptions or disease-related terms, making them different from those used in Tables 5/A26. Consequently, the same test cases receive different prediction scores, leading to differences in the reported diagnostic metrics.Coverage of liver specialist annotations.Regarding the liver tumor category, the lesion annotations used for specialist training cover multiple pathological subtypes rather than being limited to a single cancer type. The combined liver tumor category in MedVL-CT69K includes liver cysts, liver cancer, and liver metastases, all of which are covered by the specialist training labels. Moreover, the specialist training data also include other liver lesions, such as hemangiomas and focal nodular hyperplasia (FNH).For clarity, we have added a brief explanation in the revised Appendix C.4.1 of the task-specific prompting protocols and the correspondence between the combined liver-category labels and the specialist training annotations.
Table R5: Task-specific prompts for pancreatic and gastric lesion diagnosis.