acceptodds
Withdrawn from ICLR 2027

Diagnosis by Composition: Detecting and Localizing Unseen Diseases by Composing Learned Radiological Findings

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Deep models for radiological diagnosis are typically trained to predict disease labels directly from images, which limits them to diseases seen during training and makes their decisions hard to interpret in radiologists' terms. Radiology, however, proceeds from visual findings to disease through published guidelines that make this decision rule explicit, interpretable, and reproducible. We introduce diagnosis by composition (DbC), a representation learning framework that encodes this findings-first logic. DbC learns a vocabulary of radiological findings by aligning image features with the text embeddings of the findings, each finding represented as a dense map over the scan. A disease is defined by a set of required findings derived from its guidelines, and its diagnosis map is a fixed, parameter-free composition of the corresponding finding maps. Because the finding-to-disease mapping is derived from guidelines rather than learned, DbC is interpretable and auditable by design, can extend to new diseases by specifying their findings without retraining, and can diagnose unseen diseases by composing their findings. On multi-phase liver CT, DbC detects and localizes unseen diseases, rare diseases from a different institution, and an external cohort significantly better than all comparators, including models that align images with disease names. DbC also improves detection and localization in the fully supervised setting and remains superior across training-set sizes. Although demonstrated on liver CT, DbC applies wherever radiological guidelines define diseases through findings.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
medical image analysis, computer vision, zero-shot learning, radiology, tumor detection and segmentation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wFdiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This manuscript is set at the intersection of medical image analysis and explainability with the main proposition to produce disease predictions not directly end to end, but by first identifying findings in medical scans and from these findings composing the diagnosis based on medical guideline-based rules. The main contribution the manuscript argues for is the concept diagnosis by composition, where findings are detected and then composed into an interpretable diagnosis, for which a multi-modal architecture design and training regimen is proposed. As additional contributions, the manuscript argues, that the proposed modeling localizes unseen diagnoses better as compared to multi-modal models from literature, which is explored on a dataset on multi-phase liver CTs. Further, it claims, that the method based on diagnosis by composition can generalize better towards rare diseases and scans from other institutions. Explorations on data-efficient training supplement the manuscript with additional experimental insights.

Critical strengths
  1. Critical strength: Considering how to integrate rich knowledge from medical guidelines into models for making them more interpretable is a valuable area of research. Why it is consequential to my overall assessment: Deep learning models, through their end-to-end learning design obfuscate how predictions form, which is a crucial limitation especially in the medical domain, where diagnosis is consequential for patient health and needs to be auditable. Integrating information in the form of medical guidelines, which represent a focused effort by the medical community for more standardized practice, serves as valuable source for providing this auditability to models.

    (a) Enriching disease datasets with attributes / findings that were confirmed by medical experts can provide a valuable resource for the medical image analysis community.

    (b) Investigating how this knowledge can be utilized into model training for more auditable predictions is an important research area.

Critical weaknesses
  1. Critical weakness: The presentation of the manuscript heavily relies on its appendix instead of presenting the methodology, main results and contributions in the main manuscript in a self-contained manner. Why it is consequential to my overall assessment: The appendix is in my assessment so heavily referenced that understanding some statements in the main body of text require consulting the appendix, thereby outsourcing figures / descriptions / tables to the appendix to circumvent page limit. This is done in so frequently that it interrupts the reading flow substantially.

Here is a full list of references to the appendix (marked in bold) in the main body of text:

  • (line 177) with the sources for each disease listed in Appendix B. The resulting finding catalog lists the M = 40 findings of the vocabulary in twelve categories, among them density, margin and the pattern of contrast enhancement in each phase (Table 5).

  • (line 187) and each of the seven diagnoses we model requires two to five findings (Table 6).

  • (line 188) positions share findings, so one vocabulary serves every disease and a finding such as low density is supervised by the lesions of every disease that requires it (Figure 4).

  • (line 194) Each training lesion is labeled with findings from three complementary sources, which its mask places on the token grid (Appendix C).

  • (line 215) with an all-zero channel for any phase that was not acquired (Appendix A),

  • (line 247) with the readout initialization and the full recipe in Appendix D.

  • (line 255) split at the patient level into 12,487 training, 1,559 validation and 1,563 test CTs (Appendix A).

  • (line 261) for every disease and as a macro mean over diseases (Appendix F)

  • (line 292) from the fully supervised checkpoint against CTs without a liver lesion (Appendix G)

  • (line 314) and each is fine-tuned from its released checkpoint on our data (Appendix E).

  • (line 409) The radiologists on our team defined each composition from published imaging criteria, as for the internal diseases and with no reference to the model or its outputs (Table 6)

  • (line 412) both regimes at localizing the lesions and at telling the diseases apart (Table 4, Appendix H).

  • (line 455) Figure 6 shows such maps on test CTs, where each composed map follows the findings specific to its disease

While details such as training setups for baselines are not required in the main manuscript, and therefore can be stated in the appendix, main qualitative figures which support the claim of the interpretability of the proposed method, information on the findings catalog or on the assignment of voxels to diseases and findings are essential to comprehend the experimental setup and proposed method.

  1. Critical weakness: The manuscript does not properly embed itself into the literature, which is most prominently expressed by the lack of relevant baselines that was compared to and lack of previously used datasets, as well as missing relevant work discussed in the related work. Why it is consequential to my overall assessment: (a) With the selected baselines and dataset the proposed method and modeling paradigm of diagnosis by composition can not suitably be assessed to be an advantageous choice. (b) Missing crucial relations to previous works and research areas falsely exaggerates the novelty of the proposed modeling paradigm.

(a) Relevant baselines are not covered, or cited: Zero-Shot Diagnosis in CT scans with considering under-represented abnormalities and a notion on interpretability with attention heatmaps has been done by Lai et al. [A]. A broad set of multi-modal approaches for zero-shot X-Ray disease classification has been investigated where similarly to the manuscript, rare diseases are a focus point for investigation [B]. Park et al. [C] introduce RadZero3D, which, similar to the presented manuscript, explore the decomposition of CT reports into finding-sentences (Section 3.1) and train a multi-modal model with this information, where the zero-shot evaluation is done on abnormality classification. The integration of knowledge about diseases, i.e., attributes and rules has been explored by Li et al [D], where multi-modal models are prompted with text of the attributes to be found for a disease and focus on interpretability. Potentially the most related, method which has not been cited or compared to is by Jiang et al [E] from ICLR 2025, which, similar to the proposed method in the manuscript, extracts findings from radiological reports, trains a multi-modal model for zero-shot segmentation and evaluates on unseen lesions. This rich work on highly related settings are not acknowledged through citations or compared to in the manuscript's experiments (e.g. on shared publicly available datasets with associated extracted attributes/findings beyond one lung disease dataset), which leaves the question whether the proposed training regimen and design choices are actually superior to prior art unanswered.

(b) The paradigm described in the paper is related to Zero-Shot Learning with class-attributes as semantic space, for example the approach Direct Attribute Prediction (DAP) [F], uses the setup where attributes (in the manuscript called findings) are predicted and subsequently combined to a classification prediction (in the manuscript disease prediction). This research area of Zero-Shot Learning with attributes has a broad research history [G]. Even more related, Zero-Shot Semantic Segmentation literature, where the output is not a class prediction but a segmentation of a class (in the manuscript disease segmentation), which also has done based on "intermediate" attributes before [H] and most related in 3D CT data [E] (particularly Section 3.1 "(2) Dissecting Reports into Descriptions of Fundamental Disease Attributes"). As these previous works show, the idea of composing a prediction on seen / unseen classes has previously been proposed for classification on computer vision datasets and has been showcased on medical segmentation as well, limiting the novelty of the proposed "diagnosis by composition" paradigm (i.e., contribution one).

References:

Baselines that seem to be relevant:

[A] Lai, Haoran, et al. "Bridged semantic alignment for zero-shot 3d medical image diagnosis." IEEE Journal of Biomedical and Health Informatics (2025).

[B] Lin, Mingquan, et al. "CXR-LT 2024: A MICCAI challenge on long-tailed, multi-label, and zero-shot disease classification from chest X-ray." Medical Image Analysis 106 (2025): 103739.

[C] Park, Jonggwon, et al. "Radzero3d: Bridging self-supervised video models and medical vision-language alignment for zero-shot chest ct interpretation." 2025 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW). IEEE, 2025.

[D] Li, Zheng, et al. "CDCLIP: An interpretable zero-shot medical image classification framework based on concept decomposition." Biomedical Signal Processing and Control 116 (2026): 109585.

[E] Jiang, Yankai, et al. "Unleashing the potential of vision-language pre-training for 3d zero-shot lesion segmentation via mask-attribute alignment." International Conference on Learning Representations. Vol. 2025. 2025.

[F] Lampert, Christoph H., Hannes Nickisch, and Stefan Harmeling. "Learning to detect unseen object classes by between-class attribute transfer." Cvpr. Vol. 1. No. 2. 2009.

[G] Xian, Yongqin, et al. "Zero-shot learning—a comprehensive evaluation of the good, the bad and the ugly." IEEE transactions on pattern analysis and machine intelligence 41.9 (2018): 2251-2265.

[H] Zhang, Runtong, et al. "Advancing zero-shot semantic segmentation through attribute correlations." Neurocomputing 594 (2024): 127829.

Questions for authors

What is the reasoning behind prompting the baseline multi-modal models exclusively with disease names, rather than providing them with the findings-text as well, e.g., with a naive ensemble/voting process, or, under utilization of a similar aggregation strategy as described in Equation (2)? This could validate whether the proposed setup with SWIN UNETR + BiomedCLIP is superior to the modeling choices regarding backbones made in SegVol, SAT, VoxTell, or, validate the design of Equation (2).

5 comments
AuthorsOpenReview ↗

Response to Reviewer wFdi (1/3)

Thank you for the careful and constructive review! We especially appreciate the complete list of appendix pointers, the specific references and the evaluation that your question proposes, which we have run. Our response spans three comments because of the length limit. W1 is addressed in this comment, W2 in comments 1 and 2, and your question in comment 3. Letters [A] to [H] follow your reference list, significance refers to the paired CT-level bootstrap used in the paper, and table, figure and section numbers refer to the submitted version.


W1. The main text relies too heavily on the appendix.

We agree, and we apologize for the effort this cost you. The submission left to the appendix content that a reader needs in order to follow the method and the results. We are restructuring the paper and revising its writing so that the main text can be read on its own, using the additional page that the revision allows. Following your assessment of what is essential, the finding catalog and the diagnostic compositions, the assignment of findings to lesions and the qualitative maps that support interpretability will be presented in the main text, while details such as training recipes remain in the appendix.


W2 (a). How does DbC relate to the works you list?

We agree that the submission did not place itself well enough in this literature, and all eight works will be cited and discussed in the revision. They fall into four groups whose works share the same difference from DbC. The table states what each group provides and what the design of DbC enables beyond it.

GroupWorksWhat the works provideWhat DbC enables beyond them
1. Scan-level zero-shot classification[A], [B], [C]A score per scan for an abnormality named in a promptThe lesions of a disease absent from training are localized, and the findings behind each diagnosis are shown
2. Description-based prompting[D]An image label from a pretrained CLIP prompted with generic visual concepts for a diseaseThe radiological findings themselves are learned and localized, so a description is verified on the lesion itself
3. Attribute-aligned lesion segmentation[E]Masks of unseen lesion types from region embeddings aligned with attribute textEvery finding has a map of its own, and each diagnosis is computed from these maps by a guideline rule
4. Attribute-based zero-shot learning[F], [G], [H]A label or a class mask for unseen classes of natural images, inferred through shared attributesEach lesion in a 3D CT is diagnosed by the clinical definition of its disease, and the diagnosis can be audited against it

Groups 1 and 2 ([A] to [D]). BrgSA [A] and RadZero3D [C] align chest CT with report text, CXR-LT 2024 [B] is a challenge on chest radiographs with image-level labels, and CDCLIP [D] decomposes a disease into generic visual concepts, such as smooth or spotty texture, with which it prompts a CLIP pretrained on natural images. All four are evaluated by a score for the whole scan or image. BrgSA and CDCLIP show heatmaps as illustrations, and the authors of RadZero3D state that voxel-level localization has not yet been demonstrated. Zero-shot means in [A] to [C] that no class labels are used, while the training scans and reports can contain the evaluated abnormalities. In DbC the radiological findings themselves are the training targets, each is predicted as a dense map, and the maps are composed at every location. DbC therefore localizes the lesions of a disease that no training CT contains, including the rare diseases of Table 3, a focus that we share with [A] and [B]. Each diagnosis also shows the findings it rests on, where a heatmap shows only where a model responded.

Comparison and datasets. Our evaluation scores every model on localizing lesions as well as on detecting diseases, so a comparator has to return a lesion map, which the works of groups 1 and 2 do not. Their principles are nevertheless part of our comparison in a controlled form. Disease-VL matches image features with disease names at every location with the encoder, data and compute of DbC, and SegVol, SAT and VoxTell are published models of this kind, each fine-tuned on our data. Comment 3 prompts all four with descriptions of the findings, as your question proposes and in the spirit of [D], and none of them reaches DbC. Malenia [E], which does return masks, could not be included because its code and model weights have not been released. We also evaluate on MCT-LTDiag, a public dataset of four-phase liver CT with lesion masks, on which DbC is significantly better than every comparator (Table 4). None of the public datasets used in the listed works provides multi-phase liver CT with lesion masks for several liver diseases.

AuthorsOpenReview ↗

Response to Reviewer wFdi (2/3)

W2 (a) (continued). How does DbC differ from Malenia [E]?

Malenia [E] is the closest of the works you list. Both methods describe lesions by properties shared across diseases and link them to diseases through a table. DbC differs in four aspects of its design, each of which serves auditability or zero-shot diagnosis.

Every finding has a map of its own. Malenia aligns the embedding of one candidate region with all eight attributes of the lesion together, so the attributes share one mask. A reader of DbC can open the map of washout or of an enhancing capsule alone and verify it on the lesion (Figure 6).

The composition is the computation. In Malenia a learned fusion produces the mask, and the knowledge table is queried afterwards to name the lesion type. A DbC diagnosis map is the parameter-free soft minimum of its required finding maps and is therefore traceable to them.

The findings are the diagnostic criteria of clinical guidelines. They include the enhancement patterns across contrast phases that separate one liver disease from another. Malenia dissects reports into general descriptors such as shape, density and surface characteristics, and reports add only supportive findings in DbC.

The zero-shot settings are complementary. Malenia transfers mainly across organs, for example to kidney tumor while tumors of other organs are in training. DbC transfers across diseases that share an organ and can share a scan, so an unseen disease must also be told apart from the seen ones.


W2 (b). What does DbC contribute beyond attribute-based zero-shot learning?

We agree that DbC stands in the tradition of attribute-based zero-shot learning (group 4), and the submission should have said so. DAP [F] predicts the attributes of an image with one classifier per attribute and infers an unseen class from them, Xian et al. [G] establish the benchmark of this field, and Zhang et al. [H] carry it to segmentation, with object parts as the shared attributes, annotated at the pixel level. Recognizing an unseen class through attributes shared with seen classes is their principle, and we do not claim it as ours.

DbC builds this principle into a findings-first representation of 3D medical images. The model learns only the findings, as dense maps in one space aligned with finding text, and each diagnosis is computed from these maps by the rule with which published guidelines define the disease. The clinical definition of a disease thus becomes the computation of the model, and three strengths follow from this design.

Every diagnosis can be audited. In group 4 the relation between attributes and classes is side information for transfer. In DbC it is the guideline definition of the disease and the computation itself, a parameter-free function of named finding maps (Equation 1), with nothing learned between the findings and the diagnosis. A radiologist can therefore read a diagnosis through its findings and check it, finding by finding, against the criteria that the clinic already uses. This is the auditability that your review expects from guideline knowledge, and DbC provides it by construction.

Unseen diseases are diagnosed and localized. DAP predicts one attribute vector per image, which would mix the findings of the diseases that one liver can hold, and [H] takes the mask of an unseen object from its parts but its class from its name. In DbC the composition alone decides the diagnosis, at every location, so a disease is scored where all of its required findings coincide. A disease that no training CT contains is reached by writing its composition, without retraining, and DbC localizes it significantly better than every comparator for all six unseen diseases (Table 2), for two rare diseases (Table 3) and on a public external cohort (Table 4). No finding has to be delineated for this, unlike the parts of [H], since finding labels are placed on lesion masks (Section 3.3).

Findings are a better target for representation learning. DAP trains separate classifiers on fixed image features. DbC trains one encoder whose features at every location are aligned with the text of the findings, and no disease is a training target. A finding is supervised by the lesions of every disease that requires it, and because all findings share one language space, a composition can name a finding that had no training example, as every unseen disease requires (Section 4.3). In the matched ablations with the same encoder, data and compute, alignment with findings is better than alignment with disease names when every disease is supervised (Table 1), localizes every unseen disease significantly better (Table 2) and remains better at every training-set size (Section 4.5).

We will add this lineage, with [F], [G] and [H], to the related work and where the framework is introduced.

AuthorsOpenReview ↗

Response to Reviewer wFdi (3/3)

Q. Why were the baselines prompted with disease names only?

We agree that the comparison is fairer when the baselines also receive the findings text, and we have run this evaluation. The submission used disease names because each baseline was trained or fine-tuned on our data with these names, which left open whether the advantage of DbC comes from the findings text itself.

Setup. Disease-VL, SegVol, SAT and VoxTell were prompted with a description of each disease that lists the required findings of its composition (Table 6) in place of its name, so that a model receives all required findings in the one prompt it takes. For HCC the description lists "low density on the non-contrast phase, non-rim arterial hyperenhancement, non-peripheral washout and enhancing capsule". No weight was changed, and the protocol is that of Tables 1 and 2. A comparator cell gives the result with disease names / with finding descriptions.

(a) Localization, Dice.

SettingSegVolSATVoxTellDisease-VLDbC
Fully supervised.055 / .033.190 / .196.096 / .084.271 / .273.313
Unseen HCC.035 / .048.215 / .259.206 / .159.249 / .019.376
Unseen ICC.065 / .070.206 / .332.292 / .273.347 / .315.394
Unseen metastasis.021 / .013.147 / .151.013 / .010.098 / .090.165
Unseen hemangioma.051 / .048.241 / .260.094 / .158.133 / .075.332
Unseen FNH.048 / .048.150 / .151.145 / .210.035 / .168.265
Unseen cyst.001 / .004.032 / .030.001 / .000.015 / .005.059

(b) Detection, AUROC.

SettingSegVolSATVoxTellDisease-VLDbC
Fully supervised.806 / .791.900 / .819.810 / .809.916 / .873.931
Unseen HCC.757 / .750.809 / .829.811 / .812.691 / .785.834
Unseen ICC.835 / .835.804 / .861.870 / .872.828 / .872.932
Unseen metastasis.731 / .695.828 / .809.724 / .723.765 / .825.846
Unseen hemangioma.764 / .796.714 / .837.807 / .813.776 / .828.839
Unseen FNH.796 / .789.511 / .738.713 / .783.532 / .706.801
Unseen cyst.734 / .730.803 / .794.775 / .769.689 / .726.809

Findings text helps some comparators on unseen diseases, but not consistently. With descriptions, SAT is higher on most unseen diseases in both localization and detection. The other comparators gain on some unseen diseases and lose on others, and with every disease supervised the detection of every comparator is lower.

No comparator reaches DbC with either prompt. Fully supervised, DbC is significantly better than every comparator prompted with descriptions in localization and detection. It also localizes each of the six unseen diseases significantly better than every one of them and detects each better, significantly so for ICC, metastasis and cyst.

Neither the backbone nor the findings text accounts for the advantage. Disease-VL shares the SwinUNETR encoder and the BiomedCLIP text encoder of DbC, with the same data and compute, and with descriptions it also receives the findings of every composition. DbC remains significantly better than it in localization and detection when fully supervised and localizes every unseen disease significantly better. SegVol, SAT and VoxTell, with their own backbones, do not close the gap with the same text either. This points to learning each finding as its own map and composing the maps with Equation (1) as the source of the advantage.

We will add these results with every metric of Tables 1 and 2 as a new appendix and refer to them in Section 4.1.


We sincerely thank you again for the careful review, the detailed references and the evaluation you proposed! We will add the new results and the revised related work to the revision.

Reviewer_8wNeOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes Diagnosis by Composition (DbC) for multi-phase liver CT. The framework predicts dense maps of radiological findings and constructs each diagnosis map by combining the required finding maps specified in a guideline-derived diagnostic composition, using a fixed, parameter-free soft minimum. Experiments compare DbC with disease-first baselines in fully supervised diagnosis, leave-one-disease-out evaluation, rare diseases, an external cohort, and data scaling.

Critical strengths
  • The diagnostic composition is explicit and easy to inspect. Each diagnosis map is constructed from named finding maps, making it possible to examine which findings contribute to a prediction. This findings-first formulation also allows new diseases to be specified through their diagnostic compositions without retraining.
Critical weaknesses
  • It remains unclear whether the finding maps capture the intended visual findings. The same required findings are assigned to every lesion of a given disease, and each of the seven diagnostic compositions includes findings required by no other composition. This makes it difficult to distinguish learning the intended findings from learning disease masks under different labels. Spatially specific findings, such as rim arterial enhancement and upstream biliary dilatation, are also assigned to the entire lesion mask. Without finding-level evaluation, and with Fig. 6 showing only required findings, the evidence for the claimed interpretability and learned finding vocabulary remains limited.
  • The source of the improvement and the distinction from related work remain unclear. DbC differs from Disease-VL in its text anchors, supervision targets, and additional report-extracted supportive findings and size labels. These changes make it difficult to attribute the gains to guideline-based composition. In the fully supervised comparison in Table 1, only AUROC shows a significant advantage over every comparator. Ablations using only required findings and replacing the fixed soft minimum with a learned composition would help establish what drives the improvement. Malenia [1] also uses mask–attribute alignment and a clinical knowledge table to associate attributes with diseases, but it is neither discussed nor included in the comparisons.

[1] Jiang, Yankai, et al. “Unleashing the Potential of Vision-Language Pre-Training for 3D Zero-Shot Lesion Segmentation via Mask-Attribute Alignment.” The Thirteenth International Conference on Learning Representations (ICLR), 2025.

Questions for authors
  1. Were any finding labels assigned through direct, lesion-by-lesion image review? Who annotated the lesion masks, and how were the disease diagnoses established?
  2. How are descriptive findings such as well-defined and ill-defined margins, low density, very high density, and water density operationally defined? In Table 5, what does “phase-invariant” mean for low density, particularly when the hemangioma composition also requires centripetal fill-in and progressive delayed enhancement?
  3. How are conflicts between required findings and report-extracted supportive findings handled? For example, what labels are assigned to a metastasis described as having a well-defined margin, when its diagnostic composition requires an ill-defined margin?
  4. Could you provide visualizations and quantitative evaluation of finding maps supervised only by reports or the size rule, such as heterogeneous density, central necrosis, and size? Do these maps reflect differences between lesions of the same disease that are documented in the reports or measured from the masks?
  5. How do performance and generalization change when training uses only required findings or when the composition is learned? How does DbC compare with Malenia?
10 comments
AuthorsOpenReview ↗

Response to Reviewer 8wNe (1/5)

Thank you for the careful and expert review! We especially appreciate how closely you examined the way our finding labels are defined and assigned, and that each concern came with a concrete analysis that could resolve it. We ran the requested analyses and ablations and report them across five comments because of the length limit. W1 is addressed in comments 1 and 2 together with Q1 and Q4, W2 and Q5 in comments 3 and 4, and Q2 and Q3 in comment 5. Brackets are 95% CIs, and significance refers to the paired CT-level bootstrap used in the paper. Table and section numbers refer to the submitted version.


W1 and Q4. Do the finding maps capture the intended findings?

We agree that the finding maps should be evaluated directly, and that the way they are supervised makes this a fair question. Every lesion receives the required findings of its disease, so on a seen disease a finding required by only one disease is labeled on exactly the lesions of that disease, and its accuracy alone cannot tell a learned finding from a disease mask under another label. We therefore report the finding-level accuracy in (a), and then two evaluations that bear on this question directly, the findings supervised only by reports or the size rule in (b) and the unseen diseases in (c). Localization is measured by token AP, the average precision over the tokens of the evaluation grid, whose chance level is the share of positive tokens. Detection is measured by the AUROC over CTs, with the maximum of a finding map as the CT score, as for the diseases in the paper.

(a) Finding-level accuracy, fully supervised. The table gives macro means over the findings of each group on the internal test set. The third group is limited to the findings with at least ten positive test CTs.

Finding groupFindingsToken APChanceAUROC
Required, shared by several diseases6.715 [.656, .775].002.924 [.910, .937]
Required, unique to one disease14.511 [.461, .563].001.919 [.907, .931]
Supervised by reports and size only10.249 [.228, .277]<.001.915 [.903, .927]

All three groups are detected reliably and localized far above chance. Among the required findings, the shared ones are the informative group. Rim arterial enhancement, for example, is labeled on the lesions of both ICC and metastasis, and well-defined margin on those of hemangioma, FNH and cyst, so no single disease mask reproduces either of them. These shared findings are localized significantly better than the unique ones.

(b) Findings supervised only by reports or the size rule (Q4). These findings test your concern most directly, because their labels differ between lesions of the same disease. A report records heterogeneous density or central necrosis for some lesions of a disease and not for others, and the size finding of a lesion follows from its mask alone, so no disease mask can reproduce any of them. The table gives the findings you name, on the internal test set.

FindingSupervised byToken APChanceAUROC
Heterogeneous densityReports.164 [.081, .266]<.001.935 [.914, .954]
Central necrosisReports.085 [.038, .154]<.001.952 [.932, .969]
Size below 1 cmSize rule.071 [.050, .095]<.001.865 [.842, .886]
Size 1 to 2 cmSize rule.220 [.183, .258]<.001.915 [.897, .931]
Size 2 to 5 cmSize rule.523 [.474, .571]<.001.964 [.953, .974]
Size above 5 cmSize rule.933 [.915, .947].002.994 [.990, .996]

All six maps are detected reliably and localized far above chance, although their labels vary between lesions of the same disease. Localization is lower for the two report findings, which are rare, and it rises with lesion size across the four size findings. This dependence on lesion size is not specific to the finding maps or to DbC. Small lesions are a known difficulty of lesion detection and segmentation in medical imaging, and in liver CT automated methods detect and segment small lesions far less reliably than large ones (Bilic et al., MedIA 2023, and Vorontsov et al., Radiology AI 2019). The same holds in our experiments, where every model in Table 2 localizes small lesions far worse than large ones. We will add the analysis of localization accuracy against lesion size for every model to the appendix, so that this trend is documented for all of them.

AuthorsOpenReview ↗

Response to Reviewer 8wNe (2/5)

W1 (continued)

(c) Finding-level accuracy on unseen diseases. Under LODO the unseen disease contributes no label at all, so a map that merely reproduced the masks of the training diseases would have no reason to respond on its lesions. For each unseen disease, we scored every required finding of its composition against the lesions of that disease, on the checkpoint that never saw it, so that the shared and the unique findings are compared on the same lesions. Tokens of other lesions are set aside. Detection separates the CTs with the unseen disease from the CTs in which no lesion carries the finding. Bold marks the significantly higher of the two groups.

Unseen diseaseToken AP, sharedToken AP, uniqueChanceAUROC, sharedAUROC, unique
HCC.483 [.428, .537].387 [.358, .410].004.786 [.763, .809].714 [.681, .747]
ICC.769 [.734, .803].678 [.642, .715].008.870 [.842, .896].938 [.928, .949]
Metastasis.681 [.626, .735].540 [.490, .589].008.885 [.867, .903].806 [.783, .828]
Hemangioma.767 [.747, .785].368 [.349, .386].011.891 [.871, .908].711 [.668, .751]
FNH.349 [.245, .449].180 [.121, .243].005.820 [.786, .852].665 [.617, .712]
Cyst.032 [.018, .062].026 [.019, .035].001.858 [.843, .872].795 [.778, .812]

Three conclusions follow. First, both groups of findings localize the lesions of every unseen disease and detect its CTs above chance, although no CT with that disease was in training. Second, the shared findings, which keep their supervision through the other diseases, are localized better than the unique findings in every unseen disease and are in general detected better as well. Third, the unique findings had no direct supervision and still localize and detect the unseen disease above chance. This is consistent with the intuition of Section 4.3 that related finding terms lie close in the language space, so that a finding without examples of its own can be matched through the findings it is related to. Cyst is the lowest of the six in localization for both groups, as it is for every model in Table 2, in line with its small lesions and the dependence on lesion size in (b).

Together, (a) to (c) show that the finding maps are more than disease masks under other labels. They are learned for findings whose labels no disease determines, they respond on the lesions of diseases that were never in training, and they carry over according to the supervision that each finding kept.

Spatially specific findings. We agree. A finding label is placed on the whole mask of its lesion, so the map of rim arterial enhancement is trained to mark the lesion that shows the rim rather than the rim itself, and upstream biliary dilatation is labeled on the lesion rather than on the dilated ducts. A finding map therefore answers which lesion shows a finding, not where within or around the lesion the finding lies. To our knowledge, no dataset of this scale provides finding-level delineations, so training at scale has to rely on lesion-level labels. We will state this as a limitation in the Discussion. It does not affect the composition, which assigns a diagnosis to a lesion as a whole and therefore needs each finding at the resolution of the lesion.


Q1. How were the finding labels, lesion masks and diagnoses obtained?

Finding labels. The report-extracted supportive findings originate from direct image review. Each report was written by a radiologist who read the CT in clinical practice and described its lesions, and the supportive findings are extracted from these descriptions (Section 3.3 and Appendix C). The required findings follow from the diagnostic composition of the lesion's disease and the size finding from its mask, so they are not assigned by reviewing each lesion. To check these labels against the image, a radiologist reviewed a random sample of 200 test lesions one by one. No lesion carried a required finding that the radiologist judged wrong, and 12 lesions showed a further finding that their labels do not name. The findings assigned from the compositions therefore hold on the individual lesions, and what the labels miss is an occasional supportive finding. We will report this check in the revision.

Lesion masks and diagnoses. The lesion masks were annotated by practicing radiologists trained in abdominal imaging, and they are the human-annotated reference standard for every model. Each mask labels its lesion with one disease of the taxonomy (Appendix A). The diagnoses were established in clinical practice in the same way. Radiologists who read abdominal CTs as part of their clinical work mark every lesion they see and assign it to a disease category, which is what they are trained to do. Both therefore come from clinicians. We will add this description to Section 4.1.

AuthorsOpenReview ↗

Response to Reviewer 8wNe (3/5)

W2 and Q5. What drives the gain over Disease-VL?

We agree that DbC and Disease-VL differ in more than the composition step. Disease-VL is the disease-first counterpart of DbC with the same encoder, data and compute, so the submitted comparison isolates findings-first from disease-first learning as a whole. We ran both ablations you proposed, and a third experiment on the text anchors, to separate its parts.

(a) DbC trained on required findings only. We retrained DbC without the report-extracted supportive findings and the size findings, on the full training data and on every LODO split, with the original recipe. The first table gives the fully supervised results on the internal test set.

MetricDisease-VLDbC, required findings onlyDbC
Dice.271.307.313
IoU.220.248.253
Coverage.274.308.314
AP50.137.159.163
AUROC.916.929.931
AP.582.651.663
Sens. at 95% spec..646.691.699

The second table gives the unseen diseases under LODO, with each cell showing Dice / AUROC.

Unseen diseaseDisease-VLDbC, required findings onlyDbC
HCC.249 / .691.357 / .813.376 / .834
ICC.347 / .828.387 / .916.394 / .932
Metastasis.098 / .765.155 / .834.165 / .846
Hemangioma.133 / .776.302 / .830.332 / .839
FNH.035 / .532.231 / .761.265 / .801
Cyst.015 / .689.052 / .791.059 / .809

Removing the supportive and size findings changes DbC only slightly, and none of the differences from the full model is significant. The required-only model remains significantly better than Disease-VL, both fully supervised and on every unseen disease. The additional finding labels therefore account for little of the gain over Disease-VL. The full tables with 95% CI will be added to the revised paper.

(b) A learned composition. We replaced the fixed soft minimum with a learned mapping from the 40 findings to the diseases, and compared this variant with DbC in the supervised setting.

ModelDiceCoverageAP50AUROCAP
Disease-VL.271.274.137.916.582
Learned composition.301.289.157.930.635
DbC.313.314.163.931.663

The fixed composition is as accurate as the learned one and significantly better in Coverage and AP, so taking the composition from the guidelines costs no accuracy in the supervised setting.

Beyond accuracy, a learned composition gives up the two properties that the fixed composition provides. The first is interpretability and auditability. A diagnostic composition is the definition of a disease that clinical guidelines already state. It is recognized by the field and is not fitted to data, so every diagnosis map can be traced to, and audited against, the findings that the guidelines require. A learned mapping is a second learned component between the findings and the diagnosis. Nothing ties it to clinical criteria, and it can link a disease to whatever combination of findings fits the training data. The second is zero-shot diagnosis. A learned mapping exists only for the diseases seen in training, so it cannot localize or detect an unseen disease, whereas the composition of any disease can be written from its guidelines and evaluated without retraining. The learned variant can therefore be compared with DbC only in the supervised setting, and none of the zero-shot results in Tables 2 to 4 can be obtained with it.

(c) Text anchors. You also list the text anchors among the differences, so we gave the comparators the same finding text. Disease-VL, SegVol, SAT and VoxTell were each prompted with every disease described by its required findings instead of its name. The finding text does not close the gap. DbC remains significantly better than Disease-VL with finding descriptions in localization and detection when fully supervised, and it localizes every unseen disease significantly better and detects it better. The same holds against the three published models. The text anchors therefore do not account for the gain over Disease-VL. We will add the complete tables for all four models to the appendix.

Together, (a) to (c) show that the improvement over Disease-VL holds without the additional finding labels, does not require a learned composition and is not explained by the finding text. It rests on learning the findings that the guidelines require and composing them by the guideline rule. We will add all three experiments to the revision.

Reviewer_6MEKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Diagnosis by Composition (DbC), a concept-bottleneck framework for 3D lesion localization and scan-level disease detection on multi-phase liver CT. Instead of aligning image tokens directly with disease names, lesions are densely supervised using a combination of guideline-derived required findings, report-extracted supportive findings, and mask-derived size findings. The authors evaluate DbC across fully supervised internal testing, leave-one-disease-out (LODO) zero-shot generalization, external rare diseases, external multi-center transfer (MCT-LTDiag), and training data scaling.

Critical strengths

The manuscript is clearly written and easy to follow. Grounding dense 3D medical image representations in explicit radiological findings and composing them via parameter-free clinical guideline rules is intuitive and well-aligned with radiological reasoning.

Critical weaknesses
  1. Lack of Finding-Level Evaluation and Confounded Ablations: The main paper evaluates final disease predictions and provides no accuracy metrics for the 40 intermediate findings. Because clinical reports mention only ~1.2 findings per lesion on average, supervision mainly comes from deterministically copying each lesion's disease label to its guideline-required findings (including disease-unique findings that are 1-to-1 identical to the disease label). Furthermore, in LODO, the soft-minimum (β=10) may suppress disease scores when disease-unique required findings, which received no positive training supervision, are predicted with low confidence. Without evaluating findings directly, it is unclear whether DbC learns genuine visual findings or merely re-encodes disease labels. Comparing DbC with Disease-VL does not isolate the compositional modeling mechanism, because DbC also benefits from additional supervision (LLM-mined report findings, mask-size labels, and expert guideline priors) and descriptive visual prompts rather than bare disease names.

  2. Baseline Failures, Missing CBM Comparators, and Unexplained Performance Breakdowns: All published baselines (SegVol, SAT, VoxTell) fail almost completely even after full supervised fine-tuning (AP50 = .000, Dice .055–.190 in Table 1), substantially underperforming the matched Disease-V baseline without an adequate explanation in the main paper. Moreover, despite discussing Concept Bottleneck Models (CBMs) and description-based VLMs as closely related approaches, no CBM or description-prompted baseline is included. In Table 2, zero-shot localization of hepatic cysts nearly fails (0.059 Dice, 0.006 AP50).

Questions for authors
  1. Finding-Level Accuracy: What is the localization and detection accuracy on the 40 radiological findings themselves (especially comparing shared vs. disease-unique required findings in LODO)?
  2. De-confounded Ablations: How does DbC perform when trained only with guideline-required findings (removing report-mined supportive findings and size labels), and how does Disease-VL perform when prompted with guideline finding descriptions instead of bare disease names?
  3. Baseline Validity and CBM Comparison: Why do SegVol, SAT, and VoxTell achieve .000 AP50 in Table 1 (how were the 4 contrast phases fed into them?), and how does DbC compare against a standard CBM baseline?
  4. Cyst Failure Analysis: Why is zero-shot localization on cysts so low (0.059 Dice)?
6 comments
AuthorsOpenReview ↗

Response to Reviewer 6MEK (1/5)

Thank you for the careful and constructive review! We especially appreciate that each question came with a concrete way to resolve it, from finding-level evaluation to de-confounded ablations and a CBM comparison. We ran the requested experiments and report them across five comments because of the length limit. W1 is addressed in comments 1 to 3, W2 in comments 4 and 5, and each of Q1 to Q4 under the weakness it belongs to. Brackets are 95% CIs, and significance refers to the paired CT-level bootstrap used in the paper. Table and section numbers refer to the submitted version.


W1 and Q1. Do the finding maps learn findings or re-encode disease labels?

We agree that the finding maps should be evaluated directly, and that the way they are supervised makes this a fair question. Every lesion receives the required findings of its disease, so on a seen disease a finding required by only one disease is labeled on exactly the lesions of that disease, and its accuracy alone cannot tell a learned finding from the disease label. We therefore report the finding-level accuracy you asked for in (a), and then the evaluation that bears on this question directly, on unseen diseases, in (b). Localization is measured by token AP, the average precision over the tokens of the evaluation grid, whose chance level is the share of positive tokens. Detection is measured by the AUROC over CTs, with the maximum of a finding map as the CT score, as for the diseases in the paper.

(a) Finding-level accuracy, fully supervised. The table gives macro means over the findings of each group on the internal test set. The third group is limited to the findings with at least ten positive test CTs.

Finding groupFindingsToken APChanceAUROC
Required, shared by several diseases6.715 [.656, .775].002.924 [.910, .937]
Required, unique to one disease14.511 [.461, .563].001.919 [.907, .931]
Supervised by reports and size only10.249 [.228, .277]<.001.915 [.903, .927]

All three groups are detected reliably and localized far above chance. Among the required findings, the shared ones are the informative group. Rim arterial enhancement, for example, is labeled on the lesions of both ICC and metastasis, and well-defined margin on those of hemangioma, FNH and cyst, so no single disease label reproduces either of them. These shared findings are localized significantly better than the unique ones. The third group is detected at a similar level, although its labels come from reports and lesion sizes and do not follow from the disease. Its token AP is lower, as expected for report findings, which are much rarer than the required ones, and for the small lesions of the two smallest size findings.

AuthorsOpenReview ↗

Response to Reviewer 6MEK (2/5)

W1 and Q1 (continued). Do the finding maps learn findings or re-encode disease labels?

(b) Finding-level accuracy on unseen diseases. Under LODO the unseen disease contributes no label at all, so a map that merely re-encoded the training diseases would have no reason to respond on its lesions. For each unseen disease, we scored every required finding of its composition against the lesions of that disease, on the checkpoint that never saw it. All findings of a disease are thereby scored on the same lesions, which makes the shared and the unique findings directly comparable. Tokens of other lesions are set aside, and a finding is scored on the CTs with the contrast phases it is defined on. Detection separates the CTs with the unseen disease from the CTs in which no lesion carries the finding. Bold marks the significantly higher of the two groups.

Unseen diseaseToken AP, sharedToken AP, uniqueChanceAUROC, sharedAUROC, unique
HCC.483 [.428, .537].387 [.358, .410].004.786 [.763, .809].714 [.681, .747]
ICC.769 [.734, .803].678 [.642, .715].008.870 [.842, .896].938 [.928, .949]
Metastasis.681 [.626, .735].540 [.490, .589].008.885 [.867, .903].806 [.783, .828]
Hemangioma.767 [.747, .785].368 [.349, .386].011.891 [.871, .908].711 [.668, .751]
FNH.349 [.245, .449].180 [.121, .243].005.820 [.786, .852].665 [.617, .712]
Cyst.032 [.018, .062].026 [.019, .035].001.858 [.843, .872].795 [.778, .812]

Three conclusions follow. First, both groups of findings localize the lesions of every unseen disease and detect its CTs above chance, although no CT with that disease was in training. Second, the shared findings, which keep their supervision through the other diseases, are localized better than the unique findings in every unseen disease and are in general detected better as well. A finding thus carries over to an unseen disease according to the supervision that it kept. Third, the unique findings had no direct supervision and still localize and detect the unseen disease above chance. This is consistent with the intuition of Section 4.3 that related finding terms lie close in the language space, so that a finding without examples of its own can be matched through the findings it is related to. Cyst is the lowest of the six in localization for both groups, as it is for every model in Table 2, in line with the lesion sizes discussed under Q4.

The labels in (a) and (b) follow from the disease and the report of each lesion, so we also had them checked against the image. A radiologist reviewed a random sample of 200 test lesions one by one. No lesion carried a finding in its composition that the radiologist judged wrong, and 12 lesions showed a further finding that their labels do not name. The required findings assigned from the disease therefore hold on the individual lesions, and what the labels miss is an occasional supportive finding.

The soft minimum under LODO. You are right that the soft minimum is strict. A token scores high for a disease only where every required finding scores high, so a required finding predicted with low confidence lowers the composed score. This is the intended logical AND, and it makes LODO demanding for DbC, since every unseen disease has at least one required finding with no direct supervision (Section 4.3). As (b) shows, these findings are localized less accurately than the shared ones and yet remain well above chance, and DbC localizes every unseen disease significantly better than every comparator (Table 2). We will add the finding-level evaluation and the radiologist's check of the labels to the revision.

AuthorsOpenReview ↗

Response to Reviewer 6MEK (3/5)

W1 and Q2. What drives the gain over Disease-VL?

We agree that DbC and Disease-VL differ in more than the composition step. Disease-VL is the disease-first counterpart of DbC with the same encoder, data and compute, so the submitted comparison isolates findings-first from disease-first learning as a whole. We ran both experiments you proposed to separate its parts.

(a) DbC trained on required findings only. We retrained DbC without the report-extracted supportive findings and the size findings, on the full training data and on every LODO split, with the original recipe. The first table gives the fully supervised results on the internal test set.

MetricDisease-VLDbC, required findings onlyDbC
Dice.271.307.313
IoU.220.248.253
Coverage.274.308.314
AP50.137.159.163
AUROC.916.929.931
AP.582.651.663
Sens. at 95% spec..646.691.699

The second table gives the unseen diseases under LODO, with each cell showing Dice / AUROC.

Unseen diseaseDisease-VLDbC, required findings onlyDbC
HCC.249 / .691.357 / .813.376 / .834
ICC.347 / .828.387 / .916.394 / .932
Metastasis.098 / .765.155 / .834.165 / .846
Hemangioma.133 / .776.302 / .830.332 / .839
FNH.035 / .532.231 / .761.265 / .801
Cyst.015 / .689.052 / .791.059 / .809

Removing the supportive and size findings changes DbC only slightly, and none of the differences from the full model is significant. The full tables with 95% CIs will be added to the paper. The required-only model remains significantly better than Disease-VL, both fully supervised and on every unseen disease. The additional finding labels therefore account for little of the gain over Disease-VL.

(b) Comparators prompted with finding descriptions. We ran this experiment for every comparator that takes a text prompt, namely Disease-VL, SegVol, SAT and VoxTell. Each trained model was prompted with every disease described by its required findings instead of its name, on its full-data checkpoint and on every LODO checkpoint. For HCC, the description reads "low density on the non-contrast phase, non-rim arterial hyperenhancement, non-peripheral washout and enhancing capsule". For space, the table below shows Disease-VL, the matched counterpart of DbC that your question names, with each cell giving Dice / AUROC. The complete tables for all four models, with every metric and its confidence interval, will be in the updated PDF.

SettingDisease-VL, disease namesDisease-VL, finding descriptionsDbC
Fully supervised.271 / .916.273 / .873.313 / .931
Unseen HCC.249 / .691.019 / .785.376 / .834
Unseen ICC.347 / .828.315 / .872.394 / .932
Unseen metastasis.098 / .765.090 / .825.165 / .846
Unseen hemangioma.133 / .776.075 / .828.332 / .839
Unseen FNH.035 / .532.168 / .706.265 / .801
Unseen cyst.015 / .689.005 / .726.059 / .809

The finding text alone does not close the gap. DbC remains significantly better than Disease-VL with finding descriptions in localization and detection, fully supervised and on average over the unseen diseases, and it localizes every unseen disease significantly better. The same conclusion holds for all four description-prompted models. DbC localizes significantly better than each of them, fully supervised and on every unseen disease, and it detects better than each of them in every setting. The advantage of DbC therefore holds whether the comparators are prompted with disease names or with finding descriptions.

The remaining difference, the guideline knowledge, is the composition itself. It cannot be removed from DbC without removing composition, since the diagnostic composition is the rule that DbC evaluates at inference. Experiments (a) and (b) hold it fixed and remove the other two differences, the additional finding labels and the prompt text. Neither accounts for the gain over Disease-VL, which points to learning the findings and composing them as its source. We will add both experiments to the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.