acceptodds
Under review as a conference paper at ICLR 2027

One Lesion, Many Views: Disentangled Vision-Language Pretraining for Multi-Dimensional Lesion Understanding

Abstract

Vision-Language Pretraining (VLP) has emerged as a powerful paradigm for medical image understanding by leveraging unstructured reports. However, existing medical VLP methods often fall short in fine-grained lesion understanding, primarily due to the inherent sparsity of lesion-specific information within free-text reports and the lack of explicit lesion-level anchoring during alignment. To address these limitations, we propose Lesion-CLIP, a disentangled VLP framework for multi-dimensional lesion understanding. First, for what to align, we introduce multi-dimensional data decoupling by decomposing 3D CT scans and radiology reports into five distinct semantic dimensions. Second, for where to align, we develop dimension-specific space decoupling by projecting image features into distinct subspaces. Third, for how to align, we design asymmetric direction decoupling to perform hierarchical dimension-level and lesion-level alignment. To this end, we curate LiLA, a pathology-calibrated 3D CT lesion dataset spanning type, size, location, phase, and attribute to facilitate fine-grained training and evaluation. Extensive experiments on LiLA and the public MSWAL dataset demonstrate that Lesion-CLIP achieves state-of-the-art performance across both discriminative and generative tasks. Notably, zero-shot Lesion-CLIP consistently outperforms established fully supervised models across all core dimensions on LiLA, achieving an absolute accuracy gain of 7.56% in lesion classification.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-Language Pretraining, Contrastive Learning, Lesion Understanding, Medical Imaging

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_aQEFOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Lesion-CLIP, a vision–language pretraining framework for understanding prelocalized 3D CT lesions across five dimensions: type, size, anatomical location, imaging phase, and descriptive attributes. It combines multi-dimensional supervision, dimension-specific visual projection heads, and an asymmetric contrastive objective with dimension-level image-to-text alignment and lesion-level text-to-image alignment. The authors also introduce LiLA, a liver-lesion dataset containing 3D masks, physical measurements, report-derived descriptions, and pathology-based label calibration where applicable. Evaluation covers prompt-based prediction without an additional trained classifier, linear probing on LiLA and the external MSWAL dataset, and lesion attribute generation. The reported improvements are supplemented by ablations, representation analyses, and qualitative examples.

Critical strengths
  1. The paper treats lesion understanding as more than a single-label classification problem. LiLA combines localized 3D annotations with complementary information about lesion type, size, location, phase, and appearance. This could support a broader range of lesion-analysis tasks than datasets containing only masks or diagnostic labels. The dataset and its curation process therefore have potential value beyond the particular alignment architecture proposed in the paper.
  2. The evaluation extends beyond an in-domain classification task to external-dataset linear probing and free-text attribute generation. The MSWAL experiments are particularly useful because they examine transfer beyond the liver-focused pretraining data. The generation experiments also include visual-input controls, providing an initial check that the model benefits from image information rather than language regularities alone. Together, these experiments offer encouraging evidence that the learned representations may be useful in several settings.
Critical weaknesses
  1. Lesion-CLIP uses multi-dimensional supervision including type, size, location, phase, and attributes, while the main baselines do not consistently receive equivalent information. As a result, it remains unclear how much of the reported improvement comes from the proposed alignment design rather than richer and better-curated supervision. The current ablations provide some evidence, but do not fully isolate the contributions of supervision quality, semantic decomposition, projection design, and alignment direction.
  2. The representation analyses show that different projection heads learn different features, but do not directly establish semantic disentanglement or the necessity of the chosen alignment directions. In addition, the 62.21% accuracy reported in Table 1 is not clearly reconciled with the approximately 59.19% weighted accuracy implied by Figure 11. Since these issues concern both the central mechanism and the headline result, they reduce confidence in the current conclusions.
Questions for authors
  1. Please specify the images, reports, and labels available during pretraining, model selection, linear probing, and generator training, and confirm that patient-level separation is maintained across all stages. Does the 1% setting restrict only the labels used to train the linear classifier? More importantly, is there a comparison that keeps the backbone, calibrated five-dimensional information, and training budget fixed while replacing the proposed alignment design with a simpler shared-projection or standard bidirectional baseline?
  2. Discuss the relationship to earlier report-derived lesion annotation and multi-task lesion analysis, such as Yan et al.’s fine-grained lesion annotation work and MULAN. Which aspects of the supervision, representation, and learning objective are substantively different? The discussion of MedCLIP and fVLM should also acknowledge their treatment of semantic false negatives, which is relevant to repeated categorical prompts in the present objective. These clarifications would help establish the methodological contribution more precisely.
  3. Please reconcile the 62.21% accuracy in Table 1 with the approximately 59.19% sample-weighted accuracy implied by Figure 11. Are the two results based on the same checkpoint, test cases, metric definition, and prediction unit? If the main table uses lesion-level multi-phase fusion while the figure evaluates individual phases, please describe the fusion procedure and state this distinction explicitly. Otherwise, the aggregate and subgroup results should be regenerated from the same prediction records.

If the authors can adequately address the above concerns in the rebuttal and clarify the relevant experimental settings and results, I would be willing to raise my score accordingly.

6 comments
AuthorsOpenReview ↗

Response to Reviewer aQEF for ICLR 2027 Submission: Lesion-CLIP (1/3)

We thank the reviewer for finding our work valuable and useful.

Q1: Stage-by-stage data and labels, patient-level separation, and a same-budget baseline that replaces the alignment design.

A: (1) What each stage consumes. LiLA (1,722 patients / 6,136 scans / 30,134 lesion patches) and MSWAL (484 / 484 / 3,424) are each partitioned 8 : 1 : 1 at the patient level before any lesion crop is extracted (Sec. 4.1), and every stage below inherits that partition.

StageDataSupervisionTest data fitted
PretrainingLiLA training splitFive-dimensional promptsNo
Model selectionLiLA validation splitPeak validation performanceNo
Zero-shotLiLA test splitNoNo
Linear probing trainingOne dataset's training splitType labelsNo
Linear probing inferenceOne dataset's test splitNoNo
Attribute generation trainingLiLA training splitAttribute descriptionsNo
Attribute generation inferenceLiLA test splitNoNo

(2) The 1% setting. It restricts only the labels available to the linear classifier. The vision encoder stays frozen and the pretraining corpus is unchanged. We have revised the corresponding descriptions in the manuscript to clarify that these percentages refer to the labeled data used for linear-probe training.

(3) The comparison described is already in Tables 5 and 6, at rows 2 and 3. Both keep the same backbone, the same five-dimensional supervision and the same fixed training budget, and differ from the full model only in the alignment design: row 2 is the shared-projection baseline, row 3 keeps dimension-specific projections but returns to the standard symmetric bidirectional objective.

Prompt / Projection / LossLiLA zero-shot type ACC / AUC (%)MSWAL 1% ACC / AUC (%)MSWAL 100% ACC / AUC (%)Params (M) / FLOPs (G) / VRAM (GB)
combinatorial / shared / symmetric22.69 / 69.8546.50 / 56.5957.75 / 84.71130.12 / 380.77 / 2.22
decoupled / shared / symmetric58.93 / 81.4849.85 / 58.1664.44 / 87.30130.12 / 380.77 / 2.22
decoupled / independent / symmetric60.38 / 82.8250.46 / 61.2869.30 / 89.62131.17 / 380.77 / 2.32
decoupled / independent / asymmetric62.21 / 84.3551.67 / 68.2870.21 / 90.51131.17 / 380.77 / 2.32

The baseline combinatorial prompt ("This is a CT patch of {type} {size} located in {location} in the {phase} phase characterized by {attribute}.") already carries all five dimensions, so label exposure is a constant across these rows rather than a variable.

(4) The supervision side is ablated in Table 1. Under the same alignment design, removing the open attribute dimension gives 57.60 zero-shot type accuracy, removing the auxiliary phase dimension 55.22, and removing both 54.95, against 62.21 with all five dimensions. So what the curated supervision is worth is reported in Table 1, and what the design is worth in Tables 5 and 6, each with the other held fixed.

AuthorsOpenReview ↗

Response to Reviewer aQEF for ICLR 2027 Submission: Lesion-CLIP (2/3)

Q2: Relationship to LeSaNet and MULAN, and discussion on false-negative treatment in MedCLIP and fVLM.

A: (1) Our annotation pipeline shares the report-grounded lesion annotation lineage of LeSaNet and MULAN, but differs in how semantic structures are represented and learned. LeSaNet [1] mines tags from bookmarked sentences in DeepLesion [2] and trains a multi-label CNN over a 171-tag ontology (115 body parts, 27 types, and 29 attributes). MULAN [3] incorporates a tagging branch into Mask R-CNN, jointly performing lesion detection, segmentation, and tagging using 185 report-mined tags supervised by weighted binary cross-entropy. Both methods rely on fixed tag ontologies with explicitly defined hierarchical and mutually exclusive label relations. In contrast, our approach organizes lesion semantics into five dimension-specific natural language descriptions, supported by heterogeneous clinical and physical annotations, and learns their representations through disentangled vision-language pretraining.

MethodLeSaNetMULANLesion-CLIP
SupervisionReport-mined labels from a fixed 171-tag ontologyReport-mined labels from a fixed 185-tag ontologyFive dimension-specific natural language descriptions with heterogeneous clinical and physical supervision
Semantic structure modelingExplicit hierarchical and mutually exclusive label relationsExplicit hierarchical and mutually exclusive label relations, with detection–tag score refinementDisentangled dimension-specific visual subspaces with shared text representations and hierarchical alignment
ReadoutMulti-label classificationJoint detection, segmentation, and multi-label classificationZero-shot text similarity, linear probing, and LLM-based attribute generation
Input2.5D lesion patches2.5D lesion patches3D lesion patches

We have also incorporated this comparison into Sec. 2.2 of the revised manuscript to better clarify the relationship between our work and these prior efforts.

(2) What Sec. 4.4 establishes, and what direction decoupling contributes. The representation analysis in Sec. 4.4 shows that the five dimension-specific subspaces exhibit distinct representational characteristics. We note that this does not imply statistical independence among the five semantic fields, and the revised manuscript has correspondingly described the five subspaces as structurally separated. More importantly, the effectiveness of our direction decoupling is independently examined in Tables 5 and 6. Comparing rows 3 and 4, we change only the alignment loss design, while keeping the supervision, projectors, and model parameters identical. This change improves the AUC on MSWAL under 1% label supervision from 61.28 to 68.28, and zero-shot accuracy on LiLA from 60.38 to 62.21. These results provide controlled empirical evidence that the direction decoupling design contributes to performance beyond merely using different projection heads.

(3) Semantic false negatives and repeated categorical prompts. We thank the reviewer for pointing us to this line of work. MedCLIP [4] replaces InfoNCE with a knowledge-guided semantic matching loss, while fVLM [5] identifies false negatives among both normal and abnormal samples and shifts the pairing from the patient level to the disease level. Both methods address semantically related but non-identical image–text pairs that would otherwise be treated as negatives, potentially introducing conflicting gradients during contrastive learning.

In our objective, the type, size, location, and phase prompts are generated using fixed templates (Appendix B.4). Therefore, repeated categorical labels produce exactly identical text embeddings: their corresponding terms in the denominator of Eq. 2 contribute additional logits in the same direction as the positive, reducing the net attractive gradient without introducing repulsion toward an unrelated semantic concept. Moreover, dimension decoupling confines each contrastive pool to a single semantic field, so a type subspace is never contrasted against holistic reports. The attribute prompts, however, are free-form narratives, for which semantically related but non-identical descriptions can still become false negatives. We do not explicitly correct for this issue and have acknowledged this limitation in Sec. 5 of the revised manuscript.

This distinction also helps interpret Table 5. In row 1, the baseline concatenates all five semantic fields into a single prompt. Consequently, two lesions differing only in size can have highly similar holistic descriptions while still being treated as negatives. This provides a plausible explanation for the poor baseline performance (22.69% type accuracy), although the effect of false negatives is not isolated from the other benefits of data decoupling in Table 5.

References:

[1] Yan, Ke, et al. "Holistic and comprehensive annotation of clinically significant findings on diverse CT images: learning from radiology reports and label ontology." 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2019.

[2] Yan, Ke, et al. "DeepLesion: automated mining of large-scale lesion annotations and universal lesion detection with deep learning." Journal of Medical Imaging 5.3 (2018): 036501-036501.

[3] Yan, Ke, et al. "MULAN: multitask universal lesion analysis network for joint lesion detection, tagging, and segmentation." International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer International Publishing, 2019.

[4] Wang, Zifeng, et al. "Medclip: Contrastive learning from unpaired medical images and text." Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.

[5] Shui, Zhongyi, et al. "Large-scale and fine-grained vision-language pre-training for enhanced ct image understanding." International Conference on Learning Representations. Vol. 2025. 2025.

Reviewer_TwSUOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Lesion-CLIP, a 3D CT representation-learning framework that combines lesion-level supervision across type, size, location, phase, and imaging attributes with dimension-specific visual projections and asymmetric dimension-level image-to-text and lesion-level text-to-image alignment, evaluated through structured prediction and attribute generation on the newly constructed LiLA dataset and cross-organ linear-probe transfer on MSWAL.

Critical strengths

1.Valuable problem and dataset contribution. The paper addresses the important problem of multi-dimensional lesion understanding and introduces LiLA by combining volumetric lesion annotations, clinically calibrated labels, and expert-reviewed report extraction, providing explicit image–text supervision for structured prediction and fine-grained attribute description.

2.Empirical support with modest overhead. Progressive ablations show that the full model improves accuracy over data decoupling alone by 3.28 percentage points on LiLA and 5.77 points on MSWAL, while generation controls support the contribution of correctly matched visual information. These improvements come with modest additional visual-inference overhead, suggesting practical value.

Critical weaknesses

1.Unclear experimental setup. The paper states that the proposed model was pretrained on liver-centric data, but the pretraining protocol remains insufficiently specified. While substantial improvements are reported on the authors’ LiLA dataset, gains on the public MSWAL dataset are smaller—for example, the AUC improvement over Merlin under the 100% data setting is only 0.29 percentage points; the unclear pretraining data and split protocol raise concerns about comparison fairness and potential data leakage.

2.Necessity of the design. The method combines five-dimensional data decoupling, dimension-specific visual projections, and asymmetric hierarchical image–text alignment during pretraining, whereas its linear-probing evaluations use only frozen visual features. Although the ablations demonstrate performance gains, the authors should further clarify how these designs improve downstream visual representations and why the additional pretraining complexity is warranted relative to simpler training objectives.

Questions for authors

1.Could the authors clarify the experimental settings for pretraining and low-data linear probing, including the data splits and supervision used at each stage? In particular, please confirm whether test patients’ images, reports, and labels were excluded from all training stages and explain how potential data leakage was prevented.

2.Could the authors explain the difference in performance gains between LiLA and MSWAL?

3.Could the authors clarify whether the existing baselines use the same five-dimensional supervision and, if necessary, include baselines with simpler training objectives but identical supervision to distinguish the gains from additional supervision versus the proposed design?

6 comments
AuthorsOpenReview ↗

Response to Reviewer TwSU for ICLR 2027 Submission: Lesion-CLIP (1/2)

We thank the reviewer for finding our work valuable and practical.

Q1: Pretraining and low-data linear probing protocol under-specified.

A: (1) Stage-by-stage protocol. LiLA (1,722 patients / 6,136 scans / 30,134 lesion patches) and MSWAL (484 / 484 / 3,424) are each partitioned 8 : 1 : 1 at the patient level (Sec. 4.1). What each stage consumes:

StageDataSupervisionTest data fitted
PretrainingLiLA training splitFive-dimensional promptsNo
Model selectionLiLA validation splitPeak validation performanceNo
Zero-shotLiLA test splitNoNo
Linear probing trainingOne dataset's training splitType labelsNo
Linear probing inferenceOne dataset's test splitNoNo
Attribute generation trainingLiLA training splitAttribute descriptionsNo
Attribute generation inferenceLiLA test splitNoNo

(2) How leakage is prevented, separately for the two datasets. LiLA acquires several scans per patient, one per contrast phase, with the delayed phase possibly absent; a lesion is counted per phase-specific patch (Sec. 3.1), so the same anatomical lesion yields 3–4 samples.

MSWAL provides one scan and one contrast phase per patient. The source release comprises "694 CT scans from 694 patients", split into 484 training and 210 test volumes [1], and since its 210 test volumes are not publicly released, we partition the 484 available ones at the patient level with a fixed seed.

In both datasets a lesion yields one crop per available phase (1 mm isotropic resampling, one 96 × 96 × 96 patch per connected component, Sec. 3.1), and preprocessing is subject-local throughout (fixed [−1000, 1000] HU window, per-scan padding value).

References:

[1] Wu, Zhaodong, et al. "Mswal: 3d multi-class segmentation of whole abdominal lesions dataset." International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer Nature Switzerland, 2025.

AuthorsOpenReview ↗

Response to Reviewer TwSU for ICLR 2027 Submission: Lesion-CLIP (2/2)

Q2: Explain the difference in performance gains between LiLA and MSWAL.

A: (1) The pretraining corpus and alignment design jointly explain the gap in gains. LiLA's type space is our pretraining target, so the probe inherits a representation already aligned to it: it gains 1.62 points from 1% to 100% labels, yet keeps the lead at every label fraction. MSWAL asks for additional organs never in our pretraining, where Merlin's generalist pretraining corpus covers more of the target distribution and abundant labels let the probe re-shape a general representation.

However, what changes on MSWAL is organ coverage, not dimension structure. Appendix E.5 extends the subspace analysis to MSWAL and finds the same layout: the five subspaces remain separated there as well, so a new organ is read along a direction pretraining has already fixed. That is why the low-label columns still favor us, while Merlin's broader corpus pays off mainly once labels are abundant.

(2) The narrow 100% margin is stable, and the results argue against a purely organ-level shortcut. Refitting the probe five times (seeds 42–46) with the encoder, partition and hyper-parameters fixed gives 90.58 ± 0.15 AUC against Merlin's 90.18 ± 0.18, ahead in 5 of 5 matched runs, with a cross-run standard deviation below one point in every column.

MethodSeed1% Data ACC (%)1% AUC (%)10% ACC (%)10% AUC (%)100% ACC (%)100% AUC (%)
Merlin4235.2664.1749.5475.6068.6990.22
Merlin4334.6563.8948.9475.3069.3090.39
Merlin4435.5664.3148.6374.1468.0989.92
Merlin4534.9564.0849.8575.7068.3990.09
Merlin4635.8764.4349.2475.4369.0090.30
MerlinMean ± Std35.26 ± 0.4864.18 ± 0.2149.24 ± 0.4875.23 ± 0.6368.69 ± 0.4890.18 ± 0.18
Lesion-CLIP4251.6768.2857.1478.7770.2190.51
Lesion-CLIP4350.7667.0957.4578.8570.8290.68
Lesion-CLIP4451.3768.2056.8478.6869.9190.37
Lesion-CLIP4551.9868.4256.2377.3171.1290.77
Lesion-CLIP4652.5869.7458.0579.8470.5290.59
Lesion-CLIPMean ± Std51.67 ± 0.6868.35 ± 0.9457.14 ± 0.6878.69 ± 0.9070.52 ± 0.4890.58 ± 0.15

Furthermore, we break down the full-label linear probe on MSWAL by class in Appendix E.3. Within a single MSWAL organ, the probe separates kidney stone (90.68 AUC), kidney cyst (90.42) and kidney tumor (80.35), so it reads lesion characteristics rather than merely reflects host-organ identity.

Q3: Do the existing baselines use the same five-dimensional supervision, and is there a baseline with a simpler training objective under identical supervision?

A: (1) The baselines do not share our five-dimensional supervision, and this is not something a comparison protocol can fix. Each competing model is taken as a frozen encoder and was pretrained on its own corpus with its own objective, so the dimension-level information of a lesion is rarely separated in its training text, while Radiomics uses privileged 3D masks. Tables 2 and 3 therefore measure how well a representation transfers under an identical evaluation protocol, and we do not present them as a controlled comparison of objectives.

(2) The controlled comparison is already in Tables 5 and 6. All four rows share the same five-dimensional supervision. Row 1 → row 2 is exactly "identical supervision, simpler objective". The same supervision is given as one combinatorial prompt in row 1 ("This is a CT patch of {type} {size} located in {location} in the {phase} phase characterized by {attribute}."), instead of one sentence per dimension (Appendix B.4). Zero-shot type accuracy goes from 22.69% to 58.93% on LiLA, and full-label probing from 57.75% to 64.44% on MSWAL, for unchanged parameters, visual inference FLOPs and VRAM.

(3) The contribution of the phase and attribute dimensions is measured separately rather than left implicit. Table 1 reports the same model without attribute supervision (57.60 ACC), without phase (55.22) and without both (54.95), against 62.21 with all five. So the additional dimensions are worth 4.61, 6.99 and jointly 7.26 points of type accuracy.

Overall, we hope the stage-by-stage protocol, the five-run comparison and the identically-supervised rows in Tables 5 and 6 address the three questions behind the score.

I thank the authors for their rebuttal, which has addressed most of my concerns. One remaining question concerns the low-data protocol: were the 1%/10% subsets sampled at the patient or lesion-patch level, and were identical subsets used across methods?

Reviewer_NPFWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Lesion-CLIP for understanding localized 3D CT lesions. It constructs a pathology-calibrated dataset and combines dimension-specific projections with asymmetric image-text alignment. Experiments show improvements in classification and attribute generation on LiLA. The main contribution is the combination of structured lesion supervision and a corresponding learning framework.

Critical strengths
  1. The lesion-focused dataset and annotation pipeline are useful. Combining calibrated types, masks, physical measurements, and radiologist-reviewed descriptions is a reasonable way to improve fine-grained medical VLP.

  2. The experiments cover classification, generation, and external generalization. Progressive ablations and shuffled-image controls provide useful evidence for the design and visual grounding, rather than relying only on the final benchmark scores.

Critical weaknesses
  1. The table 1 and 2 claims zero-shot and label-efficiency, which feels like an potential over-claim. Pre-training and downstream task data used in table 1 and 2 come from the same datasets, and type labels enter LiLA pretraining directly. In this case, I am not sure this setting can be defined as zero-shot or few training linear probing setting, because already large amount of same domain/dataset data as well as the corresponding supervisions has entered pre-training. This matters particularly because Table 5 attributes most of the overall gain to data decoupling. Authors should provide a better explanation on the experiment setting and how to define zero-shot and linear probing in this paper.

  2. The external improvement is relatively small: 1.52 accuracy points and 0.29 AUC points over Merlin on MSWAL (Table 3). Patient-level confidence intervals or run variability are needed to judge whether this ranking is reliable.

Questions for authors

Please respond to the concern in the Weakness part and provide analysis that the performance gain on LiLA and mild improvement on OOD dataset is not due to the fact that same domain training data and labels are introduced.

4 comments
AuthorsOpenReview ↗

Response to Reviewer NPFW for ICLR 2027 Submission: Lesion-CLIP (1/2)

We thank the reviewer for finding our work useful and comprehensively evaluated.

Q1: Zero-shot and label-efficiency framing — LiLA data and type labels enter pretraining.

A: (1) Zero-shot framing. We use zero-shot in the sense CT-CLIP [1] uses it: inference with the pretrained encoders alone, no task-specific parameters trained or gradient-updated for the target. CT-CLIP separates its "zero-shot inference" from its fine-tuned VocabFine/ClassFine variants, and reports "zero-shot detection capabilities on our internal validation set from CT-RATE", the corpus it was pretrained on, where it exceeds the supervised baseline on 18/18 abnormalities. Related works such as fVLM [2] and ViSD-Boost [3] inherit this convention.

Ours is exactly that setting. Mechanically, type labels enter only as templated contrastive text (Table 8), Eq. 6 is the sole objective, and no cross-entropy over the 11 types is formed. We claim nothing class-disjoint, and we have added this definition to Appendix D of the revised manuscript to make the setting explicit.

(2) Linear probing defined. Appendix D (Implementation Details) specifies it: "a single linear classifier was trained on the image latent features extracted from the frozen pretrained Lesion-CLIP model", which corresponds to the linear probing fine-tuning method (ClassFine) in CT-CLIP [1]. The 1/10/100% axis restricts only how many probe labels that classifier sees.

(3) Where the gains come from. Every row in Tables 5 and 6 is pretrained on the same LiLA corpus, with the same backbone, budget and number of epochs, and the baseline combinatorial prompt ("This is a CT patch of {type} {size} located in {location} in the {phase} phase characterized by {attribute}.") already names all 11 types, so every row sees exactly the same type labels. Label exposure is a constant across these rows rather than a variable, so it cannot explain the spread:

Prompt / Projection / LossACC (%)AUC (%)Params (M)FLOPs (G)VRAM (GB)
combinatorial / shared / symmetric22.6969.85130.12380.772.22
decoupled / shared / symmetric58.9381.48130.12380.772.22
decoupled / independent / symmetric60.3882.82131.17380.772.32
decoupled / independent / asymmetric62.2184.35131.17380.772.32

The three decoupling steps add 39.52 points of type accuracy over the baseline row on LiLA at essentially unchanged cost. We agree data decoupling accounts for most of it, yet it changes only how the same labels are written. Moreover, the supervised models of Table 1, trained on the same split with cross-entropy directly over the 11 types and three task-specific heads, reach 52.24–54.65, below our classifier-free 62.21. Therefore, the alignment design, instead of access to the type labels during pretraining, is the source of these improvements.

References:

[1] Hamamci, Ibrahim Ethem, et al. "Generalist foundation models from a multimodal dataset for 3D computed tomography." Nature Biomedical Engineering (2026): 1-19.

[2] Shui, Zhongyi, et al. "Large-scale and fine-grained vision-language pre-training for enhanced ct image understanding." International Conference on Learning Representations. Vol. 2025. 2025.

[3] Cao, Weiwei, et al. "Boosting vision semantic density with anatomy normality modeling for medical vision-language pre-training." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2025.

AuthorsOpenReview ↗

Response to Reviewer NPFW for ICLR 2027 Submission: Lesion-CLIP (2/2)

Q2: The external improvement is relatively small.

A: (1) We understand this concern. A margin of this size is most meaningfully read against run-to-run variability, and the original submission reported single runs, so we are glad to add that measurement; it also gives us the chance to separate where our advantage is decisive from where the two models are simply close.

(2) Additional experiment. We refit the linear probe on MSWAL five times (seeds 42–46) for both our model and the second-best Merlin, at 1/10/100% of the probe labels. Only the probe is refit: the pretrained encoder and its checkpoint, the patient-level split and all probe hyper-parameters are held fixed, so the runs differ solely in classifier initialization and batch order. Seed 42 reproduces Table 3 exactly. Pretraining is not repeated, as one run costs 55.7 GPU-hours.

MethodSeed1% Data ACC (%)1% AUC (%)10% ACC (%)10% AUC (%)100% ACC (%)100% AUC (%)
Merlin4235.2664.1749.5475.6068.6990.22
Merlin4334.6563.8948.9475.3069.3090.39
Merlin4435.5664.3148.6374.1468.0989.92
Merlin4534.9564.0849.8575.7068.3990.09
Merlin4635.8764.4349.2475.4369.0090.30
MerlinMean ± Std35.26 ± 0.4864.18 ± 0.2149.24 ± 0.4875.23 ± 0.6368.69 ± 0.4890.18 ± 0.18
Lesion-CLIP4251.6768.2857.1478.7770.2190.51
Lesion-CLIP4350.7667.0957.4578.8570.8290.68
Lesion-CLIP4451.3768.2056.8478.6869.9190.37
Lesion-CLIP4551.9868.4256.2377.3171.1290.77
Lesion-CLIP4652.5869.7458.0579.8470.5290.59
Lesion-CLIPMean ± Std51.67 ± 0.6868.35 ± 0.9457.14 ± 0.6878.69 ± 0.9070.52 ± 0.4890.58 ± 0.15

(3) What the five runs show. The cross-run standard deviation is below one point for both models in every column, so no ordering in Table 3 depends on a particular fit. Concretely, our lowest of five runs still exceeds Merlin's highest in five of the six columns. Across the five-run means Lesion-CLIP is ahead in all six, by +16.41 / +4.17 at 1%, +7.90 / +3.46 at 10% and +1.83 / +0.40 at 100% (ACC / AUC), and compared seed by seed it exceeds Merlin in 5 of 5 runs in every column. The narrowest of these is AUC at full supervision (90.58 ± 0.15 vs 90.18 ± 0.18), and we present it as a small but consistent lead rather than grouping it with the much wider low-data margins.

(4) Where our advantage actually lies, and why. The accuracy margin narrows monotonically with supervision, and the two models scale differently along that axis: Merlin gains 33.43 points from 1% to full labels, while Lesion-CLIP gains only 18.85. This is consistent with the subspace geometry in Sec. 4.4 and its extension to MSWAL in Appendix E.5: the dimension-specific subspaces are already separated, so a scarce-label probe only needs to calibrate a direction that the pretraining has fixed, whereas a generalist representation may require more labeled examples to establish effective linear decision boundaries. Read this way, our benefit is largest precisely where annotation budget is smallest.

Overall, we hope the protocol clarifications and the five-run comparison address both concerns behind the score.

Thank you for your response. The response partially address my concerns. I still feel that the results of zero-shot setting provide limited verifications in terms of the generalization ability of the proposed method, which is quite important for a pretraining method. Currently I lean on keeping the current rating.

AuthorsOpenReview ↗

Follow-up Response to Reviewer NPFW for ICLR 2027 Submission: Lesion-CLIP

We thank the reviewer for recognizing the value of our Lesion-CLIP and for the suggestion.

Q: The zero-shot setting provides limited evidence of generalization for a pretraining method.

A: (1) Scope of the zero-shot protocol. It fixes the label space and replaces fine-tuning with a text-only classifier, following the definition of CT-CLIP (stated in Appendix D). We adopt this protocol because clinical decision support operates within an established differential class set, and same-corpus zero-shot tables are what recent medical vision-language works report (fVLM, ICLR 2025; ViSD-Boost, ICCV 2025). It tests alignment quality and we do not claim category-level zero-shot generalization from it.

(2) Our generalization evidence is the cross-dataset transfer, which stands independently of this protocol. The encoder pretrained on LiLA is transferred to MSWAL, a public external cohort from different sites and scanners, and evaluated as a frozen backbone at 1%, 10% and 100% probe labels (Tables 3 and 6, Appendix E.5). Across five independent runs, Lesion-CLIP outperforms the strongest baseline at every label ratio, with the largest margins under the fewest labels.

We agree that while our MSWAL experiments demonstrate consistent cross-dataset transfer under different annotation budgets, extending the evaluation to additional external cohorts, including zero-shot protocols on them, is a valuable direction that we will pursue in future work. Overall, we sincerely hope the response addresses the remaining concern behind the score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.