- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Lesion-CLIP, a vision–language pretraining framework for understanding prelocalized 3D CT lesions across five dimensions: type, size, anatomical location, imaging phase, and descriptive attributes. It combines multi-dimensional supervision, dimension-specific visual projection heads, and an asymmetric contrastive objective with dimension-level image-to-text alignment and lesion-level text-to-image alignment. The authors also introduce LiLA, a liver-lesion dataset containing 3D masks, physical measurements, report-derived descriptions, and pathology-based label calibration where applicable. Evaluation covers prompt-based prediction without an additional trained classifier, linear probing on LiLA and the external MSWAL dataset, and lesion attribute generation. The reported improvements are supplemented by ablations, representation analyses, and qualitative examples.
Critical strengths
- The paper treats lesion understanding as more than a single-label classification problem. LiLA combines localized 3D annotations with complementary information about lesion type, size, location, phase, and appearance. This could support a broader range of lesion-analysis tasks than datasets containing only masks or diagnostic labels. The dataset and its curation process therefore have potential value beyond the particular alignment architecture proposed in the paper.
- The evaluation extends beyond an in-domain classification task to external-dataset linear probing and free-text attribute generation. The MSWAL experiments are particularly useful because they examine transfer beyond the liver-focused pretraining data. The generation experiments also include visual-input controls, providing an initial check that the model benefits from image information rather than language regularities alone. Together, these experiments offer encouraging evidence that the learned representations may be useful in several settings.
Critical weaknesses
- Lesion-CLIP uses multi-dimensional supervision including type, size, location, phase, and attributes, while the main baselines do not consistently receive equivalent information. As a result, it remains unclear how much of the reported improvement comes from the proposed alignment design rather than richer and better-curated supervision. The current ablations provide some evidence, but do not fully isolate the contributions of supervision quality, semantic decomposition, projection design, and alignment direction.
- The representation analyses show that different projection heads learn different features, but do not directly establish semantic disentanglement or the necessity of the chosen alignment directions. In addition, the 62.21% accuracy reported in Table 1 is not clearly reconciled with the approximately 59.19% weighted accuracy implied by Figure 11. Since these issues concern both the central mechanism and the headline result, they reduce confidence in the current conclusions.
Questions for authors
- Please specify the images, reports, and labels available during pretraining, model selection, linear probing, and generator training, and confirm that patient-level separation is maintained across all stages. Does the 1% setting restrict only the labels used to train the linear classifier? More importantly, is there a comparison that keeps the backbone, calibrated five-dimensional information, and training budget fixed while replacing the proposed alignment design with a simpler shared-projection or standard bidirectional baseline?
- Discuss the relationship to earlier report-derived lesion annotation and multi-task lesion analysis, such as Yan et al.’s fine-grained lesion annotation work and MULAN. Which aspects of the supervision, representation, and learning objective are substantively different? The discussion of MedCLIP and fVLM should also acknowledge their treatment of semantic false negatives, which is relevant to repeated categorical prompts in the present objective. These clarifications would help establish the methodological contribution more precisely.
- Please reconcile the 62.21% accuracy in Table 1 with the approximately 59.19% sample-weighted accuracy implied by Figure 11. Are the two results based on the same checkpoint, test cases, metric definition, and prediction unit? If the main table uses lesion-level multi-phase fusion while the figure evaluates individual phases, please describe the fusion procedure and state this distinction explicitly. Otherwise, the aggregate and subgroup results should be regenerated from the same prediction records.
If the authors can adequately address the above concerns in the rebuttal and clarify the relevant experimental settings and results, I would be willing to raise my score accordingly.
6 comments
Response to Reviewer aQEF for ICLR 2027 Submission: Lesion-CLIP (2/3)
Q2: Relationship to LeSaNet and MULAN, and discussion on false-negative treatment in MedCLIP and fVLM.
A: (1) Our annotation pipeline shares the report-grounded lesion annotation lineage of LeSaNet and MULAN, but differs in how semantic structures are represented and learned. LeSaNet [1] mines tags from bookmarked sentences in DeepLesion [2] and trains a multi-label CNN over a 171-tag ontology (115 body parts, 27 types, and 29 attributes). MULAN [3] incorporates a tagging branch into Mask R-CNN, jointly performing lesion detection, segmentation, and tagging using 185 report-mined tags supervised by weighted binary cross-entropy. Both methods rely on fixed tag ontologies with explicitly defined hierarchical and mutually exclusive label relations. In contrast, our approach organizes lesion semantics into five dimension-specific natural language descriptions, supported by heterogeneous clinical and physical annotations, and learns their representations through disentangled vision-language pretraining.
| Method | LeSaNet | MULAN | Lesion-CLIP |
|---|---|---|---|
| Supervision | Report-mined labels from a fixed 171-tag ontology | Report-mined labels from a fixed 185-tag ontology | Five dimension-specific natural language descriptions with heterogeneous clinical and physical supervision |
| Semantic structure modeling | Explicit hierarchical and mutually exclusive label relations | Explicit hierarchical and mutually exclusive label relations, with detection–tag score refinement | Disentangled dimension-specific visual subspaces with shared text representations and hierarchical alignment |
| Readout | Multi-label classification | Joint detection, segmentation, and multi-label classification | Zero-shot text similarity, linear probing, and LLM-based attribute generation |
| Input | 2.5D lesion patches | 2.5D lesion patches | 3D lesion patches |
We have also incorporated this comparison into Sec. 2.2 of the revised manuscript to better clarify the relationship between our work and these prior efforts.
(2) What Sec. 4.4 establishes, and what direction decoupling contributes. The representation analysis in Sec. 4.4 shows that the five dimension-specific subspaces exhibit distinct representational characteristics. We note that this does not imply statistical independence among the five semantic fields, and the revised manuscript has correspondingly described the five subspaces as structurally separated. More importantly, the effectiveness of our direction decoupling is independently examined in Tables 5 and 6. Comparing rows 3 and 4, we change only the alignment loss design, while keeping the supervision, projectors, and model parameters identical. This change improves the AUC on MSWAL under 1% label supervision from 61.28 to 68.28, and zero-shot accuracy on LiLA from 60.38 to 62.21. These results provide controlled empirical evidence that the direction decoupling design contributes to performance beyond merely using different projection heads.
(3) Semantic false negatives and repeated categorical prompts. We thank the reviewer for pointing us to this line of work. MedCLIP [4] replaces InfoNCE with a knowledge-guided semantic matching loss, while fVLM [5] identifies false negatives among both normal and abnormal samples and shifts the pairing from the patient level to the disease level. Both methods address semantically related but non-identical image–text pairs that would otherwise be treated as negatives, potentially introducing conflicting gradients during contrastive learning.
In our objective, the type, size, location, and phase prompts are generated using fixed templates (Appendix B.4). Therefore, repeated categorical labels produce exactly identical text embeddings: their corresponding terms in the denominator of Eq. 2 contribute additional logits in the same direction as the positive, reducing the net attractive gradient without introducing repulsion toward an unrelated semantic concept. Moreover, dimension decoupling confines each contrastive pool to a single semantic field, so a type subspace is never contrasted against holistic reports. The attribute prompts, however, are free-form narratives, for which semantically related but non-identical descriptions can still become false negatives. We do not explicitly correct for this issue and have acknowledged this limitation in Sec. 5 of the revised manuscript.
This distinction also helps interpret Table 5. In row 1, the baseline concatenates all five semantic fields into a single prompt. Consequently, two lesions differing only in size can have highly similar holistic descriptions while still being treated as negatives. This provides a plausible explanation for the poor baseline performance (22.69% type accuracy), although the effect of false negatives is not isolated from the other benefits of data decoupling in Table 5.
References:
[1] Yan, Ke, et al. "Holistic and comprehensive annotation of clinically significant findings on diverse CT images: learning from radiology reports and label ontology." 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2019.
[2] Yan, Ke, et al. "DeepLesion: automated mining of large-scale lesion annotations and universal lesion detection with deep learning." Journal of Medical Imaging 5.3 (2018): 036501-036501.
[3] Yan, Ke, et al. "MULAN: multitask universal lesion analysis network for joint lesion detection, tagging, and segmentation." International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer International Publishing, 2019.
[4] Wang, Zifeng, et al. "Medclip: Contrastive learning from unpaired medical images and text." Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.
[5] Shui, Zhongyi, et al. "Large-scale and fine-grained vision-language pre-training for enhanced ct image understanding." International Conference on Learning Representations. Vol. 2025. 2025.
Response to Reviewer aQEF for ICLR 2027 Submission: Lesion-CLIP (1/3)
We thank the reviewer for finding our work valuable and useful.
Q1: Stage-by-stage data and labels, patient-level separation, and a same-budget baseline that replaces the alignment design.
A: (1) What each stage consumes. LiLA (1,722 patients / 6,136 scans / 30,134 lesion patches) and MSWAL (484 / 484 / 3,424) are each partitioned 8 : 1 : 1 at the patient level before any lesion crop is extracted (Sec. 4.1), and every stage below inherits that partition.
(2) The 1% setting. It restricts only the labels available to the linear classifier. The vision encoder stays frozen and the pretraining corpus is unchanged. We have revised the corresponding descriptions in the manuscript to clarify that these percentages refer to the labeled data used for linear-probe training.
(3) The comparison described is already in Tables 5 and 6, at rows 2 and 3. Both keep the same backbone, the same five-dimensional supervision and the same fixed training budget, and differ from the full model only in the alignment design: row 2 is the shared-projection baseline, row 3 keeps dimension-specific projections but returns to the standard symmetric bidirectional objective.
The baseline combinatorial prompt ("This is a CT patch of {type} {size} located in {location} in the {phase} phase characterized by {attribute}.") already carries all five dimensions, so label exposure is a constant across these rows rather than a variable.
(4) The supervision side is ablated in Table 1. Under the same alignment design, removing the open attribute dimension gives 57.60 zero-shot type accuracy, removing the auxiliary phase dimension 55.22, and removing both 54.95, against 62.21 with all five dimensions. So what the curated supervision is worth is reported in Table 1, and what the design is worth in Tables 5 and 6, each with the other held fixed.