- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes LGCSeg to address the challenges of insufficient hard negative samples in contrastive learning and the lack of fine-grained spatial correspondence in cross-modal interactions. It leverages large language models to assist in the retrieval of semantically relevant negative samples and employs multi-dimensional vision-language interaction to enhance the accuracy of lesion localization and segmentation. The approach was experimentally validated on two medical image segmentation datasets for pulmonary infections.
Critical strengths
- The paper divides the cross-modal alignment of medical image segmentation into two dimensions: semantic discrimination ability and spatial positioning ability. The research problem is relatively clear.
- The method details and visualization of the paper are relatively sufficient, and the claims of the paper are supported to a certain extent through reasonable experimental design and result display.
Critical weaknesses
- In medical image-text segmentation tasks, multiple text descriptions may refer to different details of the same infected region; therefore, they do not necessarily constitute "negative samples" in the true sense. Treating these samples as negatives in DITSC could potentially hinder the learning process. Although the paper’s title explicitly highlights "LLM-Guided Contrastive Learning," the current ablation studies primarily validate the utility of the two contrastive loss components rather than the unique, indispensable role of the LLM guidance itself, failing to substantiate the independent contribution of the LLM.
- As noted in Table 1, some methods were retrained by the authors, while others’ results were taken directly from existing literature. Since these results were not derived from a consistent training and evaluation environment, this limits the assessment of comparison fairness and the stability of the reported performance gains. Furthermore, the experiments are limited to only two lung-related datasets, representing a rather narrow scope.
Questions for authors
- In Figure 7, many candidates exhibit high similarity to the original text; how did the authors determine that these qualify as genuine negative samples for the learning objective? Furthermore, is it possible to provide a direct comparison between the LLM-guided approach and other baselines, such as standard semantic retrieval or the use of randomly sampled negative examples from the corpus?
- Please clarify whether the results cited in the main table and those obtained from re-training are based on the same data splits and metric definitions. Additionally, are the reported performance gains consistent across multiple experimental runs? If feasible, could the authors also evaluate the model's generalization capabilities across other modalities or on different target datasets?
6 comments
Dear Reviewer,
Thank you once again for the time you have devoted to our submission and for your thoughtful feedback. In our response, we have carefully addressed your comments and provided additional experiments, analyses, and supporting evidence to clarify the key issues raised.
We would greatly appreciate it if you could review our response when time permits. We welcome any further questions or feedback.
We look forward to continuing this discussion and thank you once again for your time and consideration.
Sincerely,
The Authors
Thank you to the authors for supplementing the ablation experiment. The LLM guided gain on MosMedData+has certain value. However, the issue of false negative samples remains unresolved. The significant differences in reproducing RefSegformer precisely indicate doubts about cross paper comparability, and a unified protocol is needed to retrain key baselines or clearly limit the scope of comparison. Fixed random seeds only ensure repeatability and cannot verify the stability of performance. Overall, I tend to maintain the original rating.
Response to Reviewer T4B4: Clarifications and Additional Evidence
Response Critical Weaknesses 1 && Questions 1:
LLM-guided selection of hard negative samples utilizes candidates that are semantically similar to the positive examples (with similar infection regions) but differ in key local spatial features (with different details). In cross-modal learning, such negative examples require the model to identify finer-grained differences in order to distinguish between them. For example, the positive sample is: “infection areas in the middle and lower lungs.” Negative sample that significantly alter the lesion: “no lung infection,” and the model might distinguish the two based solely on the coarse-grained information of “whether there is an infection,” without necessarily understanding the specific lesion location. LGCSeg selects a negative sample (In Fig. 7) : “infection in the lower lungs.” The model focuses on whether lesions are present in the middle regions, rather than relying solely on coarse semantics. Furthermore, in terms of optimizing contrastive representations, these hard negative samples with subtle differences also exhibit a larger gradient in scoring levels:
Li=−logexp(si+/τ)+∑jexp(sij−/τ)exp(si+/τ)
Let si+/sij− denote the similarity between the image and positive/negative sample, respectively. The partial derivative of the similarity score with respect to the negative sample is:
∂sij−∂Li=τpij−
pij− denotes the matching probability assigned by the model to the negative sample. This suggests that easily distinguishable, low-similarity negative samples (i.e., substantially different negatives) typically yield small probabilities and gradients. In contrast, negative samples that are confused with positive samples and have high similarity (negative samples with similar infection regions but different details), the gradients in probability are typically large. Therefore, the optimization process will more strongly drive down the matching scores for hard negative samples, thereby promoting fine-grained image-text alignment.
Methods that randomly sampled negative examples from the corpus or retrieve data directly without using an LLM not only fail to enhance contrastive learning but also disrupt the alignment space, thereby reducing overall segmentation performance. The experimental analysis described above is included in Section 4.3, “Ablation Experiments,” of the paper. The results in Supplementary Table 1 are as follows:
Response Critical Weaknesses 2 && Questions 2:
In the original paper, ARSegNet, LViT, ReCLMIS, and the Baseline were retrained. To ensure a fair comparison, we locally trained and updated methods such as TransUNet, SwinUnet, UCTransNet, and TGANet. Although we have updated these results in Table 1, this does not affect the proposed method and final experimental conclusions. Due to time and hardware constraints, during the reproduction of the remaining methods, we found significant discrepancies between our results and those reported in the reference papers for LViT (Li et al., 2023b) and ReCLMIS (Huang et al., 2024). For example, RefSegformer achieved Dice and miou scores of 59.12 and 47.12 on QaTa-COV19, whereas the results in the reference papers were 84.09 and 75.48. Therefore, UNet, UNet++, ConVIRT, CLIP, and RefSegformer still refer to the results in the LViT, ReCLMIS, and STPNet papers.
It is worth noting that our method, as well as the datasets, dataset splitting schemes, and evaluation metrics used in our experiments, are strictly consistent with those in the reference papers LViT (Li et al., 2023b) and ReCLMIS (Huang et al., 2024). Therefore, the experimental results are comparable and do not affect the experimental results or conclusions of our proposed method.
To ensure a fair evaluation of the effectiveness of each module, the experiment used a fixed random seed, ensuring that the results remained consistent across repeated runs. Under this fixed random seed, we demonstrated that each innovative module delivers a performance improvement.
The aim is to evaluate the effectiveness of cross-modal segmentation and the proposed mechanism for the specific task of segmenting lung infections. Experiments on two datasets validate this task within the context of lung infection scenarios. We are currently actively seeking new public datasets for experimentation; however, due to time and equipment constraints, we will do our best to conduct these experiments. Should there be any progress in our experiments, we will update the paper and upload the results as soon as possible.