acceptodds
Withdrawn from ICLR 2027

Negative Semantics Drive Expert Specialization in Semi-Supervised Medical Image Segmentation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent semi-supervised medical image segmentation methods have increasingly explored clinical text to alleviate the scarcity of pixel-level annotations. However, existing methods primarily use positive semantics to describe desired predictions, while overlooking information about which visual patterns should be rejected. This paper shows that structured negative semantics can provide explicit signals for learning complementary visual experts. Building on this insight, RCEA-MoE structures negative descriptions into disease, location, and extent factors to spe- cialize heterogeneous visual experts. Positive–negative semantic contrasts induce factor-specific spatial corrections, which are adaptively aggregated by an image– positive-text router, while an exponential moving average (EMA) teacher pro- vides confidence-aware supervision for unlabeled images. In a controlled ablation on QaTa-COV19 with 25% labeled training data, RCEA-MoE achieves 90.84% Dice and 83.22% mIoU, improving over the single-expert visual baseline by 3.87 and 6.27 percentage points, respectively. These findings highlight the potential of combining structured negative guidance, heterogeneous experts, and reliability- aware learning for label-efficient medical image segmentation. Code is available at: https://anonymous.4open.science/r/RCEA-MoE-91D1.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Semi-supervised learning · Medical image segmentation · VLM · Text-Guided Segmentation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_rRWQOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

Paper tackles text-guided image segmentation problem and propose RCEA-MoE. RCEA-MoE, use negative clinical description into disease, location and extend factor along with the positive description. Paper introduce Reliability-Gated Contrastive Expert Attention (RCEA) and use contrast between positive and negative text features and uses spatial gates to control feature updates. An image and positive-text router combines the experts (disease, spatial and extent) and an EMA teacher provides pseudo-label.

Critical strengths

Paper addresses a relevant problem of limited availability of annotated data and propose a solution to improve the task (segmentation) using additional textual description (negative clinical description).

Critical weaknesses
  1. I think paper would require a comprehensive evaluation and ablation to justify the claims. There many components introduced, motivation and impact needs to be validated in the results section.
  2. There are many issues with the presentation. Method section is convoluted and does not clearly give the overall Visual tokens are defined as U in figure 2 and used as V in the text, PE is defined in the caption but not present in the figure 2. There is notional overloading at places. Section 3 will require major rewritten to improve the readability.

Minor comment:

  1. There is no code in the anonymous link shared in the paper. Paper does not provide enough information for reproducibility. Literatures reported results for the baseline, were they trained in the same manner as the proposed method?
Questions for authors
  1. The paper does not provide the contribution of each clinical description to improve the final segmentation (numerical grounding will help). How did you decide to provide the additional text as a negative clinical information rather than addition the same information as part of the positive text? Is there an understanding regarding what will happen in this context?
  2. What is the single expert visual baseline? Is it ConvNeXt-Tiny with only positive clinical description? Also in the ablation, one should change one component at a time to clearly state the contribution of that component.
  3. Is there any reason for choosing ConvNext-tiny? How will the results change if we use some other model for visual feature extractor?
  4. Which language model did you use to generated the negative descriptions? Did you or any clinician checked the LLM generated negative clinical descriptions? What will happen in the case if any of the descriptions is misleading? How do you filter that?
  5. How does this method be applied to other general tasks? How do you decide upon the clinical descriptors?
  6. There are many losses used in the paper, how do you decide upon the weighting factor of each one of them? Can you provide the additional overhead in-terms of parameters and inference time? Is there an understanding by keeping the batch-average routing weights uniform will help?
  7. Numbers of TeViD does not match with the paper, on QaTa-COVID19 they provide 86.37 and authors have quoted 88.23. Did you reimplemented the baseline? Implementation details regarding the baselines should be provided in the paper.
Reviewer_q9dhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary
  • The paper investigates how negative clinical semantics can provide complementary supervision for semi-supervised medical image segmentation.
  • It proposes RCEA-MoE, which uses disease-, location-, and extent-related negative descriptions to guide heterogeneous visual experts through contrastive attention, adaptive routing, and teacher–student learning.
  • Evaluations on two pulmonary lesion segmentation datasets demonstrate improvements over existing image-only and text-guided methods under 25% and 50% labeled-data settings.
Critical strengths
  • The use of structured negative clinical semantics to guide expert specialization offers a potentially useful extension to existing positive-text-guided segmentation methods.
  • The method achieves competitive improvements on both evaluated datasets, particularly under limited annotation budgets, with ablation studies supporting the overall framework.
Critical weaknesses
  • Insufficient Evidence for the Benefits of Negative Semantics: The current ablation study does not adequately isolate the contribution of negative semantic guidance. In Table 2, factorized negative text, RCEA, adaptive routing, and auxiliary supervision are introduced simultaneously. Consequently, it remains unclear whether the improvements arise specifically from negative semantics or simply from additional text conditioning and architectural complexity.
  • Unclear Effectiveness of Semantic-Driven Expert Specialization: The correspondence between disease, location, and extent negatives and the three visual experts is manually predefined rather than learned. However, the paper provides limited evidence that this design actually induces distinct and complementary expert behaviors. Moreover, the heterogeneous experts alone improve Dice by only 0.12 points, while the two specialization objectives jointly contribute just 0.48 points. These results leave the necessity of the proposed specialization mechanism insufficiently justified.
  • Unclear Practical Availability of Text Prompts: The method requires an image-specific positive report and three negative descriptions during both training and inference. The negative reports are generated offline with language-model assistance, but their reliability and practical availability at inference are not sufficiently investigated.
Questions for authors
  • Can the authors compare the proposed method with a positive-text-only MoE, randomly generated negatives, and simpler positive–negative contrastive learning under identical architectural settings?
  • Can the authors evaluate alternative or randomly permuted negative-factor assignments, separately ablate the specialization losses, and visualize expert-specific responses to verify meaningful semantic specialization?
  • How does the method perform on medical segmentation tasks and with incomplete or inaccurate clinical descriptions?
Reviewer_7hR2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on how to leverage clinical text to provide more effective semantic guidance for semi-supervised medical image segmentation. To address this, the authors propose the RCEA-MoE framework. This framework decomposes negative semantics into three distinct factors, namely disease, location, and extent, and assigns a dedicated visual expert to each. Visual features are then refined through positive-negative semantic contrast and spatial gating, with the outputs from the various experts subsequently fused in an adaptive manner. The approach is validated through experimental results.

Critical strengths

1.The paper moves beyond traditional positive text based guidance by further exploring how to leverage negative semantics to inform the model about which visual information does not align with the target. This shift from relying solely on positive semantic matching to employing a joint discriminative approach using both positive and negative semantics is a commendable contribution.

2.The authors decompose negative semantics into three distinct factors, namely disease, location, and extent, and map each to a corresponding visual expert, thereby attempting to establish a link between these semantic factors and the learning of visual features.

Critical weaknesses
  1. The authors posit that the three experts learn, respectively, the appearance of the disease, its anatomical context, and the boundaries of the lesions. However, this division of labor is primarily achieved through the network architecture and loss functions; it is not sufficiently demonstrated whether the three experts have truly developed distinct discriminative capabilities.

  2. The authors derive corrected features by subtracting negative semantic responses from positive ones, employing a gating mechanism to control the extent of this correction. Yet, the mere existence of a discrepancy between positive and negative responses does not necessarily imply that such differences correspond to valid evidence of a lesion or to erroneous regions. Furthermore, the evaluation metrics used for comparative methods must be calculated consistently to ensure a fair assessment.

Questions for authors
  1. Can the authors present the independent prediction results from different experts, their respective feature responses, and an analysis of how each expert’s predictions respond to negative semantics?

  2. Can the authors demonstrate the changes in spatial responses before and after the correction, or provide evidence that the confidence level of the gate prediction genuinely reflects the reliability of the semantic correction? Additionally, could the authors clarify the specific calculation methods for the evaluation metrics used in the paper, and confirm whether these methods are consistent with those employed by all the baseline approaches?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.