acceptodds
Under review as a conference paper at ICLR 2027

LLM-Guided Contrastive Learning for Cross-Modal Medical Image Segmentation

Abstract

Multimodal information has shown great potential in medical image segmentation, particularly when textual semantic cues are effectively exploited. However, existing cross-modal medical image segmentation methods often suffer from coarse cross-modal fusion and alignment due to two factors: (i) a lack of challenging negative examples for contrastive optimization, and (ii) the neglect of spatial correspondences between textual semantics and local image regions. To this end, we propose a cross-modal medical image segmentation framework via large language model (LLM)-guided contrastive learning (LGCSeg), which enhances representation learning and modality alignment. LGCSeg leverages a multimodal LLM to generate textual descriptions of potential infection regions in input images, which serve as semantic references for identifying relevant hard negative samples from a domain-specific medical text corpus. These samples drive contrastive learning, enhancing the discriminative power of visual-language representations. Furthermore, a multidimensional cross-modal interaction block is introduced to enable semantic interaction between visual and textual features along global, horizontal and vertical spatial dimensions. This design achieves both global and spatial-level alignment, facilitating accurate localization of text semantics within corresponding image regions. Experimental results demonstrate that LGCSeg outperforms the base model by 1.95% in Dice and 2.83% in mIoU on QaTa-COV19 (X-ray), and by 13.04% and 19.44% on MosMedData+ (CT), respectively.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Medical image segmentation, Language and text prompts, Cross-Modal contrastive network.

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_T4B4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LGCSeg to address the challenges of insufficient hard negative samples in contrastive learning and the lack of fine-grained spatial correspondence in cross-modal interactions. It leverages large language models to assist in the retrieval of semantically relevant negative samples and employs multi-dimensional vision-language interaction to enhance the accuracy of lesion localization and segmentation. The approach was experimentally validated on two medical image segmentation datasets for pulmonary infections.

Critical strengths
  1. The paper divides the cross-modal alignment of medical image segmentation into two dimensions: semantic discrimination ability and spatial positioning ability. The research problem is relatively clear.
  2. The method details and visualization of the paper are relatively sufficient, and the claims of the paper are supported to a certain extent through reasonable experimental design and result display.
Critical weaknesses
  1. In medical image-text segmentation tasks, multiple text descriptions may refer to different details of the same infected region; therefore, they do not necessarily constitute "negative samples" in the true sense. Treating these samples as negatives in DITSC could potentially hinder the learning process. Although the paper’s title explicitly highlights "LLM-Guided Contrastive Learning," the current ablation studies primarily validate the utility of the two contrastive loss components rather than the unique, indispensable role of the LLM guidance itself, failing to substantiate the independent contribution of the LLM.
  2. As noted in Table 1, some methods were retrained by the authors, while others’ results were taken directly from existing literature. Since these results were not derived from a consistent training and evaluation environment, this limits the assessment of comparison fairness and the stability of the reported performance gains. Furthermore, the experiments are limited to only two lung-related datasets, representing a rather narrow scope.
Questions for authors
  1. In Figure 7, many candidates exhibit high similarity to the original text; how did the authors determine that these qualify as genuine negative samples for the learning objective? Furthermore, is it possible to provide a direct comparison between the LLM-guided approach and other baselines, such as standard semantic retrieval or the use of randomly sampled negative examples from the corpus?
  2. Please clarify whether the results cited in the main table and those obtained from re-training are based on the same data splits and metric definitions. Additionally, are the reported performance gains consistent across multiple experimental runs? If feasible, could the authors also evaluate the model's generalization capabilities across other modalities or on different target datasets?
6 comments
AuthorsOpenReview ↗

Response to Reviewer T4B4: Clarifications and Additional Evidence

Response Critical Weaknesses 1 && Questions 1:

LLM-guided selection of hard negative samples utilizes candidates that are semantically similar to the positive examples (with similar infection regions) but differ in key local spatial features (with different details). In cross-modal learning, such negative examples require the model to identify finer-grained differences in order to distinguish between them. For example, the positive sample is: “infection areas in the middle and lower lungs.” Negative sample that significantly alter the lesion: “no lung infection,” and the model might distinguish the two based solely on the coarse-grained information of “whether there is an infection,” without necessarily understanding the specific lesion location. LGCSeg selects a negative sample (In Fig. 7) : “infection in the lower lungs.” The model focuses on whether lesions are present in the middle regions, rather than relying solely on coarse semantics. Furthermore, in terms of optimizing contrastive representations, these hard negative samples with subtle differences also exhibit a larger gradient in scoring levels:

Li=−log⁡exp⁡(si+/τ)exp⁡(si+/τ)+∑jexp⁡(sij−/τ)L_i=-\log\frac{\exp(s_i^+/\tau)}{\exp(s_i^+/\tau)+\sum_j\exp(s_{ij}^-/\tau)}

Let si+/sij−s_{i}^{+}/s_{ij}^{-} denote the similarity between the image and positive/negative sample, respectively. The partial derivative of the similarity score with respect to the negative sample is:

∂Li∂sij−=pij−τ\frac{\partial L_i}{\partial s_{ij}^-}=\frac{p_{ij}^-}{\tau}

pij−p_{ij}^{-} denotes the matching probability assigned by the model to the negative sample. This suggests that easily distinguishable, low-similarity negative samples (i.e., substantially different negatives) typically yield small probabilities and gradients. In contrast, negative samples that are confused with positive samples and have high similarity (negative samples with similar infection regions but different details), the gradients in probability are typically large. Therefore, the optimization process will more strongly drive down the matching scores for hard negative samples, thereby promoting fine-grained image-text alignment.

Methods that randomly sampled negative examples from the corpus or retrieve data directly without using an LLM not only fail to enhance contrastive learning but also disrupt the alignment space, thereby reducing overall segmentation performance. The experimental analysis described above is included in Section 4.3, “Ablation Experiments,” of the paper. The results in Supplementary Table 1 are as follows:

MethodsMosMedData+ Dice(%)↑MosMedData+ mIoU(%)↑QaTa-COV19 Dice(%)↑QaTa-COV19 mIoU(%)↑
(a) w/o Cropus76.6863.9885.1676.98
(b) Random75.8363.3685.0476.83
(c) w/o LLM-Guided74.9762.0585.0776.83
(d) LLM-Guided(our)77.3164.7785.3677.20

Response Critical Weaknesses 2 && Questions 2:

In the original paper, ARSegNet, LViT, ReCLMIS, and the Baseline were retrained. To ensure a fair comparison, we locally trained and updated methods such as TransUNet, SwinUnet, UCTransNet, and TGANet. Although we have updated these results in Table 1, this does not affect the proposed method and final experimental conclusions. Due to time and hardware constraints, during the reproduction of the remaining methods, we found significant discrepancies between our results and those reported in the reference papers for LViT (Li et al., 2023b) and ReCLMIS (Huang et al., 2024). For example, RefSegformer achieved Dice and miou scores of 59.12 and 47.12 on QaTa-COV19, whereas the results in the reference papers were 84.09 and 75.48. Therefore, UNet, UNet++, ConVIRT, CLIP, and RefSegformer still refer to the results in the LViT, ReCLMIS, and STPNet papers.

It is worth noting that our method, as well as the datasets, dataset splitting schemes, and evaluation metrics used in our experiments, are strictly consistent with those in the reference papers LViT (Li et al., 2023b) and ReCLMIS (Huang et al., 2024). Therefore, the experimental results are comparable and do not affect the experimental results or conclusions of our proposed method.

To ensure a fair evaluation of the effectiveness of each module, the experiment used a fixed random seed, ensuring that the results remained consistent across repeated runs. Under this fixed random seed, we demonstrated that each innovative module delivers a performance improvement.

The aim is to evaluate the effectiveness of cross-modal segmentation and the proposed mechanism for the specific task of segmenting lung infections. Experiments on two datasets validate this task within the context of lung infection scenarios. We are currently actively seeking new public datasets for experimentation; however, due to time and equipment constraints, we will do our best to conduct these experiments. Should there be any progress in our experiments, we will update the paper and upload the results as soon as possible.

Dear Reviewer,

Thank you once again for the time you have devoted to our submission and for your thoughtful feedback. In our response, we have carefully addressed your comments and provided additional experiments, analyses, and supporting evidence to clarify the key issues raised.

We would greatly appreciate it if you could review our response when time permits. We welcome any further questions or feedback.

We look forward to continuing this discussion and thank you once again for your time and consideration.

Sincerely,

The Authors

Thank you to the authors for supplementing the ablation experiment. The LLM guided gain on MosMedData+has certain value. However, the issue of false negative samples remains unresolved. The significant differences in reproducing RefSegformer precisely indicate doubts about cross paper comparability, and a unified protocol is needed to retrain key baselines or clearly limit the scope of comparison. Fixed random seeds only ensure repeatability and cannot verify the stability of performance. Overall, I tend to maintain the original rating.

Reviewer_Ze2aOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LGCSeg, a language-guided medical image segmentation framework that addresses insufficiently challenging negative samples and coarse spatial correspondence between visual and textual representations. The framework comprises three modules:

  1. DITSC (Dual Image–Text Selective and Contrastive) retrieves the Top-N hard negatives from a structured corpus of pulmonary infection descriptions. It ranks corpus entries by combining their feature similarities to the original and LLM-generated texts by a frozen GPT-4o mini. It performs image–text contrastive learning using negatives from both the mini-batch and the corpus.
  2. MDCMI (Multidimensional Cross-Modal Interaction) constructs three spatial representations through global, height and width dimensions of visual features to capture the left–right and upper–lower spatial relationships expressed in the text.
  3. CAC (Coordinate Attention Combined) applies coordinate attention to the encoder’s skip-connection features during decoding. LGCSeg achieves Dice/mIoU scores of 85.36%/77.20% on the QaTa-COV19 chest X-ray dataset and 77.31%/64.77% on the MosMedData+ CT dataset, outperforming all compared image-only and language-guided segmentation methods. The authors further conduct ablation studies by individually removing within-batch contrastive learning, corpus-level contrastive learning, MDCMI, and CAC, and provide visualization of segmentation results and attention.
Critical strengths
  1. The integration of LLM-generated semantics with a structured corpus offers an insightful approach to constructing hard negatives for language-guided segmentation. DITSC uses LLM to generate auxiliary semantics from original image and text, which guide hard-negative retrieval from a corpus of structured pulmonary infection descriptions. This design provides a promising strategy for using generative models to enhance medical image–text contrastive learning and solve the limitation of conventional in-batch negatives.
  2. Directional image–text interaction aligns well with the spatial semantics of pulmonary infection descriptions. Terms such as “left/right” and “upper/middle/lower” convey explicit spatial information in pulmonary infection descriptions. MDCMI constructs visual representations of global, horizontal and vertical together and integrates each with text features. This design efficiently connects directional relationships in language with the spatial organization of visual features in this task.
Critical weaknesses
  1. With the current experiments e.g., w/o Corpus, it does not establish the benefit specifically attributable to LLM-guided negative selection. The paper presents LLM-guided hard-negative selection as a core contribution, but the “w/o Corpus” ablation in Table 2 removes both the corpus-level contrastive objective and its associated negatives. Thus, it is unclear whether the gains come from adding corpus-based negatives or from using the LLM to select them. The current results do not sufficiently demonstrate that LLM guidance provides meaningful additional benefits over simpler selection strategies.
  2. Although the proposed method achieved better performance when compared to the existing methods, however, it seems that the improvement is more likely coming from a strong baseline rather than the proposed negative selection approach which can be evidenced from Table 2.
  3. It is also not clear how the original positive and negative samples are selected. In addition, the appendix describes that the LLM-generated reference descriptions were manually checked.
  4. The Gaussian-kernel bandwidth and contrastive temperature are introduced but their values are not reported, despite their direct influence on negative selection and optimization.
Questions for authors
  1. It’s not clear that why disease progression particularly useful for constructing hard negatives? The proposed LLM prompt predicts additional regions that might become infected. What makes these descriptions more informative for segmenting the current image than descriptions that directly perturb its current lesion location, count, or extent?
  2. Where LLM-guided contrastive learning helps distinguish closely related descriptions that the model otherwise confuses?
  3. What is the benefit of multidimensional cross-modal interaction? It seems that both MDCMI and CAC already incorporate directional spatial information. What additional capability comes from interacting with text separately along the horizontal and vertical dimensions?
4 comments
AuthorsOpenReview ↗

Response to Reviewer Ze2a: Clarifications and Additional Evidence

Response Weaknesses 1:

LGCSeg utilizes LLM-guided negative sample selection for contrastive learning, enhancing the discriminative ability of vision-language representations while improving segmentation performance. In contrast, methods that add random negative examples from the corpus or do not use LLMs not only fail to enhance contrastive learning but also disrupt the alignment space, thereby reducing overall segmentation performance. The experimental analysis described above is included in Section 4.3 of the paper, “Ablation Experiments.” Supplementary Table 1 presents the results of the various experiments as follows:

MethodsMosMed Dice(%)MosMed mIoU(%)QaTa Dice(%)QaTa mIoU(%)
(a) w/o Cropus76.6863.9885.1676.98
(b) Random75.8363.3685.0476.83
(c) w/o LLM-Guided74.9762.0585.0776.83
(d) LLM-Guided(our)77.3164.7785.3677.20

Response Weaknesses 2:

It is important to note that the robust foundation model is built upon the retention of blocks such as MDCMI and CAC, which is also one of the core contributions. These innovative blocks perform different functions, which is not inconsistent with the further gains brought about by the LLM-guided mechanism. Furthermore, detailed evidence has been provided regarding the independent contribution of the LLM-guided negative sample selection mechanism.

Response Weaknesses 3:

The positive examples for an image are the paired text descriptions. Batch-level negative sampling constructs negative pairs within mini-batches. In corpus-level, the LLM-guided mechanism helps select negative samples from an external corpus. We will provide more detailed descriptions of positive and negative sample selection in Section 3.4, “DITSC block,” as shown in Eq. (6): LBatch=−1B∑i=1Blog⁡exp⁡(K(Fimgi,FTi)/τ)∑j=1Bexp⁡(K(Fimgi,FTj)/τ)L_{Batch}=-\frac1B\sum_{i=1}^B\log\frac{\exp(K(F_{img_i},F_{T_i})/\tau)}{\sum_{j=1}^B\exp(K(F_{img_i},F_{T_j})/\tau)}. FimgiF_{img_i} and FTiF_{T_i} represent the paired image and text features, i.e., positive samples. Meanwhile, FimgiF_{img_i} and FTjF_{T_j} represent this image and other text features within the same batch, i.e., negative samples. Negative examples in the corpus, as shown in Eq(9): FCorpusN=TopN(WLLM⋅Km(FT,FCorpus(m)))F_{Corpus_N}=TopN(W_{LLM}\cdot K^m(F_T,F_{Corpus}^{(m)})).FCorpusNF_{Corpus_N} refers to the selected negative samples.

Response Weaknesses 4:

The temperature is fixed at τ = 0.02. The Gaussian bandwidth is calculated adaptively based on the variance of the features in the current batch.

Response Questions 1:

A key consideration in detecting disease progression is to leverage the prior reasoning capabilities of LLMs to introduce spatially correlated local changes while preserving the disease context, and thereby retrieve hard negative samples that could semantically confuse positive samples. Compared to unconstrained perturbations, this design maintains consistency in core medical semantics while concentrating fine-grained differences on segmentation-related attributes, such as lesion regions. In terms of optimizing contrastive representations, these subtly differentiated negative samples also exhibit greater gradients in score levels:

Li=−log⁡exp⁡(si+/τ)exp⁡(si+/τ)+∑jexp⁡(sij−/τ)L_i=-\log\frac{\exp(s_i^+/\tau)}{\exp(s_i^+/\tau)+\sum_j\exp(s_{ij}^-/\tau)}

Let si+s_i^+ and sij−s_{ij}^- denote the similarity between the image and positive/negative sample, respectively. The partial derivative of the similarity score with respect to the negative sample is:

∂Li∂sij−=pij−τ\frac{\partial L_i}{\partial s_{ij}^-}=\frac{p_{ij}^-}{\tau}

This suggests that for negative samples that are well distinguished by the model and have low similarity, the corresponding gradients at the probability and score levels are typically small. For negative samples that are easily confused with positive descriptions and have high similarity, the corresponding gradients at the probability and score levels are typically large, and reducing the match scores for these negative samples is driven more strongly by the optimization process.

Response Questions 2:

When conditions share a common disease background but exhibit local pathological variations, these descriptions are semantically similar to positive samples but can easily lead to confusion. In such cases, the contrastive learning optimization of the LLM-guided mechanism performs better, resulting in a model with stronger discriminative capabilities. For example, this is demonstrated by the partial derivative reasoning in the optimization of contrastive representations in Response Questions 1.

Response Questions 3:

MDCMI refines image–text alignment through directional spatial correspondences for lesion localization. Horizontal and vertical attention link positional terms such as “left/right” and “top/middle/bottom” to image regions, emphasizing lesion location and distribution. As shown in Fig. 8, the global view captures some directional information but provides limited lesion focus, whereas multidimensional interaction improves localization.

Thanks for the detailed response. I have increased my rating. but still has great concerns with regard to the improvement and with the ablation studies.

AuthorsOpenReview ↗

Response to Reviewer Ze2a: Clarifications and Additional Evidence

Response Weaknesses 1:

LGCSeg utilizes LLM-guided negative sample selection for contrastive learning, enhancing the discriminative ability of vision-language representations while improving segmentation performance. In contrast, methods that add random negative examples from the corpus or do not use LLMs not only fail to enhance contrastive learning but also disrupt the alignment space, thereby reducing overall segmentation performance. The experimental analysis described above is included in Section 4.3 of the paper, “Ablation Experiments.” Supplementary Table 1 presents the results of the various experiments as follows:

MethodsMosMed Dice(%)MosMed mIoU(%)QaTa Dice(%)QaTa mIoU(%)
(a) w/o Cropus76.6863.9885.1676.98
(b) Random75.8363.3685.0476.83
(c) w/o LLM-Guided74.9762.0585.0776.83
(d) LLM-Guided(our)77.3164.7785.3677.20

Response Weaknesses 2:

It is important to note that the robust foundation model is built upon the retention of blocks such as MDCMI and CAC, which is also one of the core contributions. These innovative blocks perform different functions, which is not inconsistent with the further gains brought about by the LLM-guided mechanism. Furthermore, detailed evidence has been provided regarding the independent contribution of the LLM-guided negative sample selection mechanism.

Response Weaknesses 3:

The positive examples for an image are the paired text descriptions. Batch-level negative sampling constructs negative pairs within mini-batches. In corpus-level, the LLM-guided mechanism helps select negative samples from an external corpus. We will provide more detailed descriptions of positive and negative sample selection in Section 3.4, “DITSC block,” as shown in Eq. (6): LBatch=−1B∑i=1Blog⁡exp⁡(K(Fimgi,FTi)/τ)∑j=1Bexp⁡(K(Fimgi,FTj)/τ)L_{Batch}=-\frac1B\sum_{i=1}^B\log\frac{\exp(K(F_{img_i},F_{T_i})/\tau)}{\sum_{j=1}^B\exp(K(F_{img_i},F_{T_j})/\tau)}. FimgiF_{img_i} and FTiF_{T_i} represent the paired image and text features, i.e., positive samples. Meanwhile, FimgiF_{img_i} and FTjF_{T_j} represent this image and other text features within the same batch, i.e., negative samples. Negative examples in the corpus, as shown in Eq(9): FCorpusN=TopN(WLLM⋅Km(FT,FCorpus(m)))F_{Corpus_N}=TopN(W_{LLM}\cdot K^m(F_T,F_{Corpus}^{(m)})).FCorpusNF_{Corpus_N} refers to the selected negative samples.

Response Weaknesses 4:

The temperature is fixed at τ = 0.02. The Gaussian bandwidth is calculated adaptively based on the variance of the features in the current batch.

Response Questions 1:

A key consideration in detecting disease progression is to leverage LLMs to introduce spatially correlated local changes while preserving the disease context, and thereby retrieve hard negative samples that could semantically confuse positive samples. Compared to unconstrained perturbations, this design maintains consistency in core medical semantics while concentrating fine-grained differences on segmentation-related attributes, such as lesion regions. In terms of optimizing contrastive representations, these subtly differentiated negative samples also exhibit greater gradients in score levels:

Li=−log⁡exp⁡(si+/τ)exp⁡(si+/τ)+∑jexp⁡(sij−/τ)L_i=-\log\frac{\exp(s_i^+/\tau)}{\exp(s_i^+/\tau)+\sum_j\exp(s_{ij}^-/\tau)}

Let si+s_i^+ and sij−s_{ij}^- denote the similarity between the image and positive/negative sample, respectively. The partial derivative of the similarity score with respect to the negative sample is:

∂Li∂sij−=pij−τ\frac{\partial L_i}{\partial s_{ij}^-}=\frac{p_{ij}^-}{\tau}

This suggests that for negative samples that are well distinguished by the model and have low similarity, the corresponding gradients at the probability and score levels are typically small. For negative samples that are easily confused with positive descriptions and have high similarity, the corresponding gradients at the probability and score levels are typically large, and reducing the match scores for these negative samples is driven more strongly by the optimization process.

Response Questions 2:

When conditions share a common disease background but exhibit local pathological variations, these descriptions are semantically similar to positive samples but can easily lead to confusion. In such cases, the contrastive learning optimization of the LLM-guided mechanism performs better, resulting in a model with stronger discriminative capabilities. For example, this is demonstrated by the partial derivative reasoning in the optimization of contrastive representations in Response Questions 1.

Response Questions 3:

MDCMI refines image–text alignment through directional spatial correspondences for lesion localization. Horizontal and vertical attention link positional terms such as “left/right” and “upper/middle/lower” to image regions, emphasizing lesion location and distribution. As shown in Fig. 8, the global view captures some directional information but provides limited lesion focus, whereas multidimensional interaction improves localization.

AuthorsOpenReview ↗

Response to Reviewer Ze2a

Thank you for reconsidering our work and raising your rating. We understand that you still have concerns regarding the scope of the ablation study. We will provide further analysis of the experiment:

Supplementary Table 1 shows the results of different experiments: (a) no corpus comparison at all; (b) using a corpus but with randomly selected negative examples; (c) without using an LLM; (d) using a corpus and an LLM to assist in selecting difficult negative examples—that is, the proposed method. The proposed method achieved the best results among the four settings on both the Dice and mIoU metrics across the two datasets. Compared to our approach’s Dice and mIoU results of 77.31 and 64.77, the random selection of negative samples on MosMedData+ yielded Dice and mIoU scores of 75.83 and 63.36, representing decreases of 1.48 and 1.41 percentage points, respectively. The methods that did not use LLMs achieved scores of 74.97 and 62.05+ on MosMedData, respectively, showing the most significant decline in performance. Importantly, compared to the approach that uses no corpus at all, both the random selection of negative samples and the method without LLM guidance actually resulted in lower scores. Not only do these approaches fail to improve the original learning performance, but they also disrupt the original alignment space, thereby reducing the overall effectiveness of the method.

This suggests that, under the experimental setup described here, simply introducing a corpus may not be sufficient to improve segmentation performance. Randomly selecting samples or selecting them without LLM guidance makes it difficult to fully ensure that the selected samples are suitable for learning the embedding space in the comparison, which may introduce training signals that are detrimental to representation alignment and thus affect the final segmentation performance. In contrast, the LLM-guided selection strategy outperformed the aforementioned alternative strategies on both datasets and had a positive effect when a corpus was used. Furthermore, in a previous response regarding the optimization of contrastive representation learning, evidence from the optimization process for these LLM-guided hard-negative samples showed that:

∂Li∂sij−=pij−τ\frac{\partial L_i}{\partial s_{ij}^-}=\frac{p_{ij}^-}{\tau}

pij−p_{ij}^{-} denotes the matching probability assigned by the model to the jth negative sample after softmax normalization between the positive sample and all candidate negative samples, specifically:

pij−=exp⁡(sij−/τ)exp⁡(si+/τ)+∑kexp⁡(sik−/τ)p_{ij}^{-}=\frac{\exp\left(s_{ij}^{-}/\tau\right)}{\exp\left(s_{i}^{+}/\tau\right)+\sum_{k}\exp\left(s_{ik}^{-}/\tau\right)}

Therefore, for negative samples that the model can distinguish well and that have low similarity (i.e., negative samples that have undergone significant changes), the gradients in terms of probability and score are typically small. Conversely, for negative samples that are easily confused with positive samples and have high similarity (where the infected areas are similar but the details differ), the gradients at the probability and score levels are typically larger. Consequently, the optimization process drives a more significant reduction in the matching scores for these negative samples, thereby promoting the model’s ability to achieve fine-grained image-text alignment.

Overall, LGCSeg utilizes LLMs to generate potential medical text descriptions based on input images, using these descriptions as semantic references to identify challenging negative examples from a domain-specific medical text corpus. These examples drive contrastive learning, thereby enhancing the discriminative capability of vision-language representations.

Thank you once again for your professional review. We hope you will consider our work again, and we look forward to actively pursuing further research in the future.

Reviewer_cz9wOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LGCSeg segments lung infections from an image and a medical description. Before training, a frozen multimodal LLM generates descriptions of possible additional infection regions. These are used to retrieve negative texts from a structured pulmonary-infection corpus, adding a corpus contrastive loss to the usual in-batch image–text loss. The segmentation network combines a U-shaped architecture with global, horizontal, and vertical cross-modal interactions (MDCMI) and coordinate-attention decoder blocks (CAC). The paper evaluates the method on QaTa-COV19 X-rays and MosMedData+ CT images, reporting Dice scores of 85.36% and 77.31%, with component ablations and segmentation visualizations.

Critical strengths

The use of horizontal and vertical pooling makes sense for the descriptions used here: left/right and upper/middle/lower lung carry spatial information that the network can exploit. Table 2 supports the usefulness of MDCMI and CAC, with a particularly large effect from CAC on MosMedData+. The full model also has the best Dice and mIoU scores on both datasets in Table 1 and exceeds each single-component ablation. These results provide reasonable preliminary evidence for the combined architecture on the two lung-infection benchmarks.

Critical weaknesses
  1. Table 2 does not tell me what the LLM adds. Removing the entire corpus-contrastive component reduces Dice by 0.20 percentage points on QaTa-COV19 and 0.63 points on MosMedData+, but there is no comparison with random corpus negatives or retrieval using only the original description. Those controls are needed to separate LLM guidance from the benefit of adding corpus negatives. Removing CAC, which incorporates the existing coordinate-attention mechanism, produces a much larger 8.43-point Dice drop on MosMedData+. The improvements over the strongest listed comparators are 0.50 and 1.13 Dice points on the two datasets, and no repeated-run uncertainty is reported. The results therefore give stronger support to the complete architecture than to the LLM-guided selection mechanism emphasized in the paper.

  2. I also could not reconstruct the attention operation from Section 3.3. Both queries and keys are said to come from text, while values come from image features with global, horizontally pooled, or vertically pooled sequence lengths. Equation (4) requires matching key and value sequence lengths, but the necessary projection or matching is not described. It is also unclear how the three branches become a spatial feature map again. Equation (3) uses a softmax mask output, while the loss section specifies one category and gives segmentation losses that do not clearly match that output. These could be notation mistakes; they do not establish that the implementation is wrong. The paper needs to state the actual computation consistently.

Questions for authors
  1. How does LLM-guided retrieval compare with random corpus negatives and retrieval using only the original description, keeping the architecture, corpus, number of negatives, and training budget fixed? Repeated-run results for these comparisons and the corpus ablation would help distinguish a stable contribution from training variation.

  2. Please give the query, key, and value shapes for each MDCMI branch and explain how the branch outputs are combined into spatial features. What binary segmentation activation and loss were actually implemented, given Eq. (3) and the stated single category? The Gaussian bandwidth and contrastive temperature are also needed.

  3. Is the positive description excluded from the corpus candidates? In Fig. 7, the original description includes middle/lower regions in both lungs, while several selected negatives describe subsets of those regions. How are equivalent or overlapping descriptions handled, and what makes these examples inconsistent with the image rather than compatible alternatives?

  4. Where does the original description used at test time come from, and does producing it require the test segmentation mask? The appendix gives image-pair counts and refers to the LViT split, but does not explicitly establish whether MosMedData+ patients and CT volumes are disjoint across partitions. Please clarify that split and whether all hyperparameters were selected using validation data only.

2 comments
AuthorsOpenReview ↗

Response to Reviewer cz9w: Clarifications and Additional Evidence

Response Critical Weaknesses 1 && Questions 1:

LGCSeg uses LLM guidance to select hard negative samples and drive contrastive learning, thereby enhancing vision-language representation learning and alignment. To further validate the additional contributions provided by the LLM, we have included a comparison with methods that randomly select negative examples from the corpus and those that use original descriptions for retrieval, and we have added the experimental analysis to Section 4.3 “Ablation Experiments” of our paper. However, these methods lack the selection of hard negative examples; not only do they fail to improve the learning performance, but they also disrupt the original alignment space, thereby reducing the overall effectiveness. Supplementary Table 1 presents the results of these experiments as follows:

MethodsMosMedData+ Dice(%)MosMedData+ mIoU(%)QaTa-COV19 Dice(%)QaTa-COV19 mIoU(%)
(a) w/o Cropus76.6863.9885.1676.98
(b) Random75.8363.3685.0476.83
(c) w/o LLM-Guided74.9762.0585.0776.83
(d) LLM-Guided(our)77.3164.7785.3677.20

The experiments in Table 2 are designed to measure the conditional contributions of each component within the entire system. The performance degradation observed after removing the CAC is not inconsistent with the LLM-guided mechanism. Furthermore, the unique contribution of the LLM-guided mechanism has been demonstrated.

To fairly validate the effectiveness of the modules, the experiments used a fixed random seed. Under these conditions, we demonstrates that each innovative module leads to improvements.

Response Critical Weaknesses 2 && Questions 2:

We detail the attention operations in Section 3.3, including length matching for Query, Key and Value, and spatial feature map reconstruction by the three branches. Additionally, we have improved the notation for the softmax function in Eq. (3) by replacing it with the sigmoid. Some of the details are as follows:

Given visual features X∈RB×C×H×WX\in\mathbb{R}^{B\times C\times H\times W} and text features T∈RB×L×CT\in\mathbb{R}^{B\times L\times C}. XX are averaged and pooled horizontally and vertically, respectively, and then unfolded together with the global branch to yield: Sh∈RB×H×CS_h\in\mathbb{R}^{B\times H\times C};Sw∈RB×W×CS_w\in\mathbb{R}^{B\times W\times C};Sg∈RB×HW×CS_g\in\mathbb{R}^{B\times HW\times C}. The cross-attention is computed as: $Q_r=S_rW_Q,K=TW_K,V=TW_V,A_r=\operatorname{softmax}_L\left(\frac{Q_rK^\top}{\sqrt{d_k}}\right),\quad O_r=A_rV, $ with tensor dimensions: QrB×Nr×dk K⊤B×dk×L→ArB×Nr×L;ArB×Nr×L VB×L×dv→OrB×Nr×dv\overset{B\times N_r\times d_k}{Q_r}\ \overset{B\times d_k\times L}{K^\top} \to \overset{B\times N_r\times L}{A_r};\quad \overset{B\times N_r\times L}{A_r}\ \overset{B\times L\times d_v}{V} \to \overset{B\times N_r\times d_v}{O_r}. Broadcasting recovers spatial resolution: vertical: B×H×C⟶B×C×H×1⟶B×C×H×WB\times H\times C\longrightarrow B\times C\times H\times1\longrightarrow B\times C\times H\times W; global: B×HW×C⟶B×C×HW⟶B×C×H×WB\times HW\times C\longrightarrow B\times C\times HW\longrightarrow B\times C\times H\times W.

The temperature is fixed at τ=0.02\tau = 0.02. The Gaussian bandwidth is calculated adaptively based on the variance of the features in the current batch.

Response Questions 3:

Before performing retrieval and negative sample selection, exclude the positive samples from the candidate corpus.

Regarding the meaning of the partially overlapping descriptions in Fig. 7. The rationale for selecting difficult negative samples in this paper is to utilize descriptions that are semantically similar to positive descriptions but differ in key regions. It can confuse positive samples, and this difference requires the model to learn finer-grained image-text correspondences. For example, a positive description: “Bilateral pulmonary infection, middle lower left lung and middle lower right lung.” Negative description: “No lung infection.” The model might be able to distinguish between the two using only coarse-grained information about “whether there is an infection,” without necessarily needing to understand the location of the lesion. If the negative description were “lower left lung and lower right lung.” the model would focus on whether lesions exist in the middle regions during the contrastive learning process, rather than relying solely on shared semantics such as “lung infection.”

Response Questions 4:

The original descriptions used during testing are those paired with the images, which are provided as complete paired data in the QaTa-COV19 and MosMedData+ datasets; therefore, there is no need to test segmentation masks. The data sources and splitting methods used in this study fully comply with the requirements specified in LViT (Li et al., 2023b) and ReCLMIS (Huang et al., 2024). Furthermore, we ensured that there was no overlap between the slice data in the training, validation, and test sets. Hyperparameters were selected using the validation set.

Dear Reviewer,

Thank you once again for the time you have devoted to our submission and for your thoughtful feedback. In our response, we have carefully addressed your comments and provided additional experiments, analyses, and supporting evidence to clarify the key issues raised.

We would greatly appreciate it if you could review our response when time permits. We welcome any further questions or feedback.

We look forward to continuing this discussion and thank you once again for your time and consideration.

Sincerely, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.