- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
This manuscript is set at the intersection of medical image analysis and explainability with the main proposition to produce disease predictions not directly end to end, but by first identifying findings in medical scans and from these findings composing the diagnosis based on medical guideline-based rules. The main contribution the manuscript argues for is the concept diagnosis by composition, where findings are detected and then composed into an interpretable diagnosis, for which a multi-modal architecture design and training regimen is proposed. As additional contributions, the manuscript argues, that the proposed modeling localizes unseen diagnoses better as compared to multi-modal models from literature, which is explored on a dataset on multi-phase liver CTs. Further, it claims, that the method based on diagnosis by composition can generalize better towards rare diseases and scans from other institutions. Explorations on data-efficient training supplement the manuscript with additional experimental insights.
Critical strengths
-
Critical strength: Considering how to integrate rich knowledge from medical guidelines into models for making them more interpretable is a valuable area of research. Why it is consequential to my overall assessment: Deep learning models, through their end-to-end learning design obfuscate how predictions form, which is a crucial limitation especially in the medical domain, where diagnosis is consequential for patient health and needs to be auditable. Integrating information in the form of medical guidelines, which represent a focused effort by the medical community for more standardized practice, serves as valuable source for providing this auditability to models.
(a) Enriching disease datasets with attributes / findings that were confirmed by medical experts can provide a valuable resource for the medical image analysis community.
(b) Investigating how this knowledge can be utilized into model training for more auditable predictions is an important research area.
Critical weaknesses
- Critical weakness: The presentation of the manuscript heavily relies on its appendix instead of presenting the methodology, main results and contributions in the main manuscript in a self-contained manner. Why it is consequential to my overall assessment: The appendix is in my assessment so heavily referenced that understanding some statements in the main body of text require consulting the appendix, thereby outsourcing figures / descriptions / tables to the appendix to circumvent page limit. This is done in so frequently that it interrupts the reading flow substantially.
Here is a full list of references to the appendix (marked in bold) in the main body of text:
-
(line 177) with the sources for each disease listed in Appendix B. The resulting finding catalog lists the M = 40 findings of the vocabulary in twelve categories, among them density, margin and the pattern of contrast enhancement in each phase (Table 5).
-
(line 187) and each of the seven diagnoses we model requires two to five findings (Table 6).
-
(line 188) positions share findings, so one vocabulary serves every disease and a finding such as low density is supervised by the lesions of every disease that requires it (Figure 4).
-
(line 194) Each training lesion is labeled with findings from three complementary sources, which its mask places on the token grid (Appendix C).
-
(line 215) with an all-zero channel for any phase that was not acquired (Appendix A),
-
(line 247) with the readout initialization and the full recipe in Appendix D.
-
(line 255) split at the patient level into 12,487 training, 1,559 validation and 1,563 test CTs (Appendix A).
-
(line 261) for every disease and as a macro mean over diseases (Appendix F)
-
(line 292) from the fully supervised checkpoint against CTs without a liver lesion (Appendix G)
-
(line 314) and each is fine-tuned from its released checkpoint on our data (Appendix E).
-
(line 409) The radiologists on our team defined each composition from published imaging criteria, as for the internal diseases and with no reference to the model or its outputs (Table 6)
-
(line 412) both regimes at localizing the lesions and at telling the diseases apart (Table 4, Appendix H).
-
(line 455) Figure 6 shows such maps on test CTs, where each composed map follows the findings specific to its disease
While details such as training setups for baselines are not required in the main manuscript, and therefore can be stated in the appendix, main qualitative figures which support the claim of the interpretability of the proposed method, information on the findings catalog or on the assignment of voxels to diseases and findings are essential to comprehend the experimental setup and proposed method.
- Critical weakness: The manuscript does not properly embed itself into the literature, which is most prominently expressed by the lack of relevant baselines that was compared to and lack of previously used datasets, as well as missing relevant work discussed in the related work. Why it is consequential to my overall assessment: (a) With the selected baselines and dataset the proposed method and modeling paradigm of diagnosis by composition can not suitably be assessed to be an advantageous choice. (b) Missing crucial relations to previous works and research areas falsely exaggerates the novelty of the proposed modeling paradigm.
(a) Relevant baselines are not covered, or cited: Zero-Shot Diagnosis in CT scans with considering under-represented abnormalities and a notion on interpretability with attention heatmaps has been done by Lai et al. [A]. A broad set of multi-modal approaches for zero-shot X-Ray disease classification has been investigated where similarly to the manuscript, rare diseases are a focus point for investigation [B]. Park et al. [C] introduce RadZero3D, which, similar to the presented manuscript, explore the decomposition of CT reports into finding-sentences (Section 3.1) and train a multi-modal model with this information, where the zero-shot evaluation is done on abnormality classification. The integration of knowledge about diseases, i.e., attributes and rules has been explored by Li et al [D], where multi-modal models are prompted with text of the attributes to be found for a disease and focus on interpretability. Potentially the most related, method which has not been cited or compared to is by Jiang et al [E] from ICLR 2025, which, similar to the proposed method in the manuscript, extracts findings from radiological reports, trains a multi-modal model for zero-shot segmentation and evaluates on unseen lesions. This rich work on highly related settings are not acknowledged through citations or compared to in the manuscript's experiments (e.g. on shared publicly available datasets with associated extracted attributes/findings beyond one lung disease dataset), which leaves the question whether the proposed training regimen and design choices are actually superior to prior art unanswered.
(b) The paradigm described in the paper is related to Zero-Shot Learning with class-attributes as semantic space, for example the approach Direct Attribute Prediction (DAP) [F], uses the setup where attributes (in the manuscript called findings) are predicted and subsequently combined to a classification prediction (in the manuscript disease prediction). This research area of Zero-Shot Learning with attributes has a broad research history [G]. Even more related, Zero-Shot Semantic Segmentation literature, where the output is not a class prediction but a segmentation of a class (in the manuscript disease segmentation), which also has done based on "intermediate" attributes before [H] and most related in 3D CT data [E] (particularly Section 3.1 "(2) Dissecting Reports into Descriptions of Fundamental Disease Attributes"). As these previous works show, the idea of composing a prediction on seen / unseen classes has previously been proposed for classification on computer vision datasets and has been showcased on medical segmentation as well, limiting the novelty of the proposed "diagnosis by composition" paradigm (i.e., contribution one).
References:
Baselines that seem to be relevant:
[A] Lai, Haoran, et al. "Bridged semantic alignment for zero-shot 3d medical image diagnosis." IEEE Journal of Biomedical and Health Informatics (2025).
[B] Lin, Mingquan, et al. "CXR-LT 2024: A MICCAI challenge on long-tailed, multi-label, and zero-shot disease classification from chest X-ray." Medical Image Analysis 106 (2025): 103739.
[C] Park, Jonggwon, et al. "Radzero3d: Bridging self-supervised video models and medical vision-language alignment for zero-shot chest ct interpretation." 2025 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW). IEEE, 2025.
[D] Li, Zheng, et al. "CDCLIP: An interpretable zero-shot medical image classification framework based on concept decomposition." Biomedical Signal Processing and Control 116 (2026): 109585.
[E] Jiang, Yankai, et al. "Unleashing the potential of vision-language pre-training for 3d zero-shot lesion segmentation via mask-attribute alignment." International Conference on Learning Representations. Vol. 2025. 2025.
[F] Lampert, Christoph H., Hannes Nickisch, and Stefan Harmeling. "Learning to detect unseen object classes by between-class attribute transfer." Cvpr. Vol. 1. No. 2. 2009.
[G] Xian, Yongqin, et al. "Zero-shot learning—a comprehensive evaluation of the good, the bad and the ugly." IEEE transactions on pattern analysis and machine intelligence 41.9 (2018): 2251-2265.
[H] Zhang, Runtong, et al. "Advancing zero-shot semantic segmentation through attribute correlations." Neurocomputing 594 (2024): 127829.
Questions for authors
What is the reasoning behind prompting the baseline multi-modal models exclusively with disease names, rather than providing them with the findings-text as well, e.g., with a naive ensemble/voting process, or, under utilization of a similar aggregation strategy as described in Equation (2)? This could validate whether the proposed setup with SWIN UNETR + BiomedCLIP is superior to the modeling choices regarding backbones made in SegVol, SAT, VoxTell, or, validate the design of Equation (2).
5 comments
Response to Reviewer wFdi (2/3)
W2 (a) (continued). How does DbC differ from Malenia [E]?
Malenia [E] is the closest of the works you list. Both methods describe lesions by properties shared across diseases and link them to diseases through a table. DbC differs in four aspects of its design, each of which serves auditability or zero-shot diagnosis.
Every finding has a map of its own. Malenia aligns the embedding of one candidate region with all eight attributes of the lesion together, so the attributes share one mask. A reader of DbC can open the map of washout or of an enhancing capsule alone and verify it on the lesion (Figure 6).
The composition is the computation. In Malenia a learned fusion produces the mask, and the knowledge table is queried afterwards to name the lesion type. A DbC diagnosis map is the parameter-free soft minimum of its required finding maps and is therefore traceable to them.
The findings are the diagnostic criteria of clinical guidelines. They include the enhancement patterns across contrast phases that separate one liver disease from another. Malenia dissects reports into general descriptors such as shape, density and surface characteristics, and reports add only supportive findings in DbC.
The zero-shot settings are complementary. Malenia transfers mainly across organs, for example to kidney tumor while tumors of other organs are in training. DbC transfers across diseases that share an organ and can share a scan, so an unseen disease must also be told apart from the seen ones.
W2 (b). What does DbC contribute beyond attribute-based zero-shot learning?
We agree that DbC stands in the tradition of attribute-based zero-shot learning (group 4), and the submission should have said so. DAP [F] predicts the attributes of an image with one classifier per attribute and infers an unseen class from them, Xian et al. [G] establish the benchmark of this field, and Zhang et al. [H] carry it to segmentation, with object parts as the shared attributes, annotated at the pixel level. Recognizing an unseen class through attributes shared with seen classes is their principle, and we do not claim it as ours.
DbC builds this principle into a findings-first representation of 3D medical images. The model learns only the findings, as dense maps in one space aligned with finding text, and each diagnosis is computed from these maps by the rule with which published guidelines define the disease. The clinical definition of a disease thus becomes the computation of the model, and three strengths follow from this design.
Every diagnosis can be audited. In group 4 the relation between attributes and classes is side information for transfer. In DbC it is the guideline definition of the disease and the computation itself, a parameter-free function of named finding maps (Equation 1), with nothing learned between the findings and the diagnosis. A radiologist can therefore read a diagnosis through its findings and check it, finding by finding, against the criteria that the clinic already uses. This is the auditability that your review expects from guideline knowledge, and DbC provides it by construction.
Unseen diseases are diagnosed and localized. DAP predicts one attribute vector per image, which would mix the findings of the diseases that one liver can hold, and [H] takes the mask of an unseen object from its parts but its class from its name. In DbC the composition alone decides the diagnosis, at every location, so a disease is scored where all of its required findings coincide. A disease that no training CT contains is reached by writing its composition, without retraining, and DbC localizes it significantly better than every comparator for all six unseen diseases (Table 2), for two rare diseases (Table 3) and on a public external cohort (Table 4). No finding has to be delineated for this, unlike the parts of [H], since finding labels are placed on lesion masks (Section 3.3).
Findings are a better target for representation learning. DAP trains separate classifiers on fixed image features. DbC trains one encoder whose features at every location are aligned with the text of the findings, and no disease is a training target. A finding is supervised by the lesions of every disease that requires it, and because all findings share one language space, a composition can name a finding that had no training example, as every unseen disease requires (Section 4.3). In the matched ablations with the same encoder, data and compute, alignment with findings is better than alignment with disease names when every disease is supervised (Table 1), localizes every unseen disease significantly better (Table 2) and remains better at every training-set size (Section 4.5).
We will add this lineage, with [F], [G] and [H], to the related work and where the framework is introduced.
Response to Reviewer wFdi (3/3)
Q. Why were the baselines prompted with disease names only?
We agree that the comparison is fairer when the baselines also receive the findings text, and we have run this evaluation. The submission used disease names because each baseline was trained or fine-tuned on our data with these names, which left open whether the advantage of DbC comes from the findings text itself.
Setup. Disease-VL, SegVol, SAT and VoxTell were prompted with a description of each disease that lists the required findings of its composition (Table 6) in place of its name, so that a model receives all required findings in the one prompt it takes. For HCC the description lists "low density on the non-contrast phase, non-rim arterial hyperenhancement, non-peripheral washout and enhancing capsule". No weight was changed, and the protocol is that of Tables 1 and 2. A comparator cell gives the result with disease names / with finding descriptions.
(a) Localization, Dice.
| Setting | SegVol | SAT | VoxTell | Disease-VL | DbC |
|---|---|---|---|---|---|
| Fully supervised | .055 / .033 | .190 / .196 | .096 / .084 | .271 / .273 | .313 |
| Unseen HCC | .035 / .048 | .215 / .259 | .206 / .159 | .249 / .019 | .376 |
| Unseen ICC | .065 / .070 | .206 / .332 | .292 / .273 | .347 / .315 | .394 |
| Unseen metastasis | .021 / .013 | .147 / .151 | .013 / .010 | .098 / .090 | .165 |
| Unseen hemangioma | .051 / .048 | .241 / .260 | .094 / .158 | .133 / .075 | .332 |
| Unseen FNH | .048 / .048 | .150 / .151 | .145 / .210 | .035 / .168 | .265 |
| Unseen cyst | .001 / .004 | .032 / .030 | .001 / .000 | .015 / .005 | .059 |
(b) Detection, AUROC.
| Setting | SegVol | SAT | VoxTell | Disease-VL | DbC |
|---|---|---|---|---|---|
| Fully supervised | .806 / .791 | .900 / .819 | .810 / .809 | .916 / .873 | .931 |
| Unseen HCC | .757 / .750 | .809 / .829 | .811 / .812 | .691 / .785 | .834 |
| Unseen ICC | .835 / .835 | .804 / .861 | .870 / .872 | .828 / .872 | .932 |
| Unseen metastasis | .731 / .695 | .828 / .809 | .724 / .723 | .765 / .825 | .846 |
| Unseen hemangioma | .764 / .796 | .714 / .837 | .807 / .813 | .776 / .828 | .839 |
| Unseen FNH | .796 / .789 | .511 / .738 | .713 / .783 | .532 / .706 | .801 |
| Unseen cyst | .734 / .730 | .803 / .794 | .775 / .769 | .689 / .726 | .809 |
Findings text helps some comparators on unseen diseases, but not consistently. With descriptions, SAT is higher on most unseen diseases in both localization and detection. The other comparators gain on some unseen diseases and lose on others, and with every disease supervised the detection of every comparator is lower.
No comparator reaches DbC with either prompt. Fully supervised, DbC is significantly better than every comparator prompted with descriptions in localization and detection. It also localizes each of the six unseen diseases significantly better than every one of them and detects each better, significantly so for ICC, metastasis and cyst.
Neither the backbone nor the findings text accounts for the advantage. Disease-VL shares the SwinUNETR encoder and the BiomedCLIP text encoder of DbC, with the same data and compute, and with descriptions it also receives the findings of every composition. DbC remains significantly better than it in localization and detection when fully supervised and localizes every unseen disease significantly better. SegVol, SAT and VoxTell, with their own backbones, do not close the gap with the same text either. This points to learning each finding as its own map and composing the maps with Equation (1) as the source of the advantage.
We will add these results with every metric of Tables 1 and 2 as a new appendix and refer to them in Section 4.1.
We sincerely thank you again for the careful review, the detailed references and the evaluation you proposed! We will add the new results and the revised related work to the revision.
Response to Reviewer wFdi (1/3)
Thank you for the careful and constructive review! We especially appreciate the complete list of appendix pointers, the specific references and the evaluation that your question proposes, which we have run. Our response spans three comments because of the length limit. W1 is addressed in this comment, W2 in comments 1 and 2, and your question in comment 3. Letters [A] to [H] follow your reference list, significance refers to the paired CT-level bootstrap used in the paper, and table, figure and section numbers refer to the submitted version.
W1. The main text relies too heavily on the appendix.
We agree, and we apologize for the effort this cost you. The submission left to the appendix content that a reader needs in order to follow the method and the results. We are restructuring the paper and revising its writing so that the main text can be read on its own, using the additional page that the revision allows. Following your assessment of what is essential, the finding catalog and the diagnostic compositions, the assignment of findings to lesions and the qualitative maps that support interpretability will be presented in the main text, while details such as training recipes remain in the appendix.
W2 (a). How does DbC relate to the works you list?
We agree that the submission did not place itself well enough in this literature, and all eight works will be cited and discussed in the revision. They fall into four groups whose works share the same difference from DbC. The table states what each group provides and what the design of DbC enables beyond it.
Groups 1 and 2 ([A] to [D]). BrgSA [A] and RadZero3D [C] align chest CT with report text, CXR-LT 2024 [B] is a challenge on chest radiographs with image-level labels, and CDCLIP [D] decomposes a disease into generic visual concepts, such as smooth or spotty texture, with which it prompts a CLIP pretrained on natural images. All four are evaluated by a score for the whole scan or image. BrgSA and CDCLIP show heatmaps as illustrations, and the authors of RadZero3D state that voxel-level localization has not yet been demonstrated. Zero-shot means in [A] to [C] that no class labels are used, while the training scans and reports can contain the evaluated abnormalities. In DbC the radiological findings themselves are the training targets, each is predicted as a dense map, and the maps are composed at every location. DbC therefore localizes the lesions of a disease that no training CT contains, including the rare diseases of Table 3, a focus that we share with [A] and [B]. Each diagnosis also shows the findings it rests on, where a heatmap shows only where a model responded.
Comparison and datasets. Our evaluation scores every model on localizing lesions as well as on detecting diseases, so a comparator has to return a lesion map, which the works of groups 1 and 2 do not. Their principles are nevertheless part of our comparison in a controlled form. Disease-VL matches image features with disease names at every location with the encoder, data and compute of DbC, and SegVol, SAT and VoxTell are published models of this kind, each fine-tuned on our data. Comment 3 prompts all four with descriptions of the findings, as your question proposes and in the spirit of [D], and none of them reaches DbC. Malenia [E], which does return masks, could not be included because its code and model weights have not been released. We also evaluate on MCT-LTDiag, a public dataset of four-phase liver CT with lesion masks, on which DbC is significantly better than every comparator (Table 4). None of the public datasets used in the listed works provides multi-phase liver CT with lesion masks for several liver diseases.