acceptodds
Withdrawn from ICLR 2027

S3-Adapt: Grounded Semantic Transfer between Foundation Models for Medical Image Segmentation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

In medical segmentation cascades built around independently encoded image-text similarity and decoder prompts, language can guide mask prediction without shaping the segmentation encoder's spatial representations. We introduce S3-Adapt, a framework that connects a pre-aligned vision-language model with a promptable segmentation foundation model through Spatial Grounding, Semantic Transfer, and Selective Correction. Reciprocal Semantic Grounding integrates query-local visual context and reciprocal region-token associations into the source visual encoder, with dense grounding supervision shaping query-conditioned spatial features. Hierarchical Cross-Foundation Semantic Transfer projects these features into multiple stages of the segmentation encoder and jointly conditions gated residual updates on source semantics and the current target state. Point, continuous-mask, and text prompts complement this representation-level transfer to form a reference prediction. Evidence-Conditioned Selective Correction then combines local image evidence with grounding and prediction context to produce a single signed correction gate. The gate controls the direction and bounded magnitude of a logit update while the reference pathway remains fixed. The original parameters of both pretrained encoder backbones remain frozen. Experiments on four public medical image segmentation datasets demonstrate the effectiveness of S3-Adapt and competitive segmentation performance with only 1.97M task-adapted parameters. Source code is provided in the supplementary material.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
medical image segmentation, vision-language models, semantic transfer, parameter-efficient adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_rKcAOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes S³-Adapt, a framework that transfers grounded visual-language representations from BiomedCLIP into MedSAM2 for medical image segmentation. It combines reciprocal semantic grounding (RSG), hierarchical cross-foundation semantic transfer (HCST), and evidence-conditioned selective correction (ECSC). The original encoder backbones remain frozen, with 1.97M task-adapted parameters. Experiments cover four medical segmentation datasets.

Critical strengths
  • Transferring grounded semantic features directly into the segmentation encoder is a meaningful alternative to relying only on text-derived prompts.
  • This work evaluates four datasets across different imaging modalities and includes useful component, placement, and boundary-quality ablations.
Critical weaknesses
  • Several reproduced results are substantially lower than those reported in the original papers. It raises serious fairness concerns. For example, HiMix reports 91.17/79.44 Dice on QaTa-COV19/MosMedData+, but Table 1 reports 84.75/71.53. Similar discrepancies appear for MMI-UNet, EviVLM, and other baselines. Some parameter counts also differ from the original implementations. The authors need to explain these differences and provide detailed reproduction settings. Without this, the claimed superiority over existing methods is difficult to assess.
  • The method also has relatively high FLOPs. Please report inference latency, memory usage, and full training cost.
  • Component ablations are useful, but the study lacks parameter-matched adapters and alternative transfer methods, making it unclear whether gains stem from semantic transfer or added adaptation capacity.
  • The method relies on image-specific descriptions of location and shape, but does not test incorrect, incomplete, generic, or contradictory text. The paper should clarify how such detailed descriptions would be obtained at inference.
Questions for authors
  • Why are several baseline results substantially lower than those in the original papers? Were the same architectures, pretrained weights, data splits, input resolutions, and text prompts used?
  • Does HCST outperform simpler parameter-matched adapters or feature-fusion alternatives?
Reviewer_XwoDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This submission introduces S³-Adapt, applied on the backbone with BiomedCLIP, the source VLM, and MedSAM2. It designs 3 main components: RSG inserts reciprocal region–token attention into BiomedCLIP blocks 7 and 9, with dense grounding supervision through the frozen projection. HCST injects the resulting grounded features S into Hiera blocks 0, 2, 9 and 11 as gated, target-state-conditioned residuals. In a second stage, ECSC learns a bounded tanh logit correction from local image evidence, uncertainty, and grounding–reference disagreement. The grounding map yields point, continuous-mask and text prompts for a LoRA-adapted decoder. With 1.97M trainable parameters, it reports the best Dice and mIoU on the benchmarks.

Critical strengths
  1. Transferring at the representation level into the SFM encoder, rather than only through prompts, is a reasonable and reasonable positioned alternative.
  2. The trainable footprint is small and both backbones stay frozen.
  3. Metrics are broad and computed in a unified pipeline. Component and placement ablations are included, and code is provided.
Critical weaknesses
  1. The writing and presentation need improvements. For example, in the introduction, I can not clearly get the motivations and what challenges are resolved behind the three key designs. Section 3 then describes the modules one by one without tying them back to that motivation, so it is hard to see why each is designed this way, e.g., reciprocal association vs. standard cross-attention, or target-conditioned gating vs. additive fusion. Concrete values (feature dimensions, selected layers such as blocks 7/9 and 0/2/9/11) should move from the method to the experiments. The text is also dense and jargon-heavy, and OHA and FSM are used in the main text without definition.
  2. The text prompts may encode ground-truth spatial information. In the examples shown in Figure 5, text prompts are per-image descriptions of count, size, shape and coarse location derived from the annotations, e.g., "unilateral, one infected area, lower right lung". At test time this is privileged label information that is unavailable in practice. The authors should consider adding more ablations such as (1) class-name-only text; (2) text shuffled across images; (3) location fields removed; and even (4) no text. Without these, it is unclear how much of RSG and grounding comes from understanding the image versus reading the answer from the text.
  3. Comparison problem: One key confound is encoder adaptation versus "semantic transfer." The ablation baseline (Table 2) keeps the MedSAM2 encoder frozen, so any encoder-side adapter would be expected to help. HCST's +3.5/+3.7 Dice may simply reflect added encoder capacity. Additional controls are needed: for example, a parameter-matched encoder LoRA or adapter on MedSAM2 without source features; HCST fed with shuffled, zero, or noise S; HCST fed with vanilla BiomedCLIP features (no RSG, no text).
  4. Missing baselines. The closest competitors are discussed in related work but never compared: MedSAM and MedSAM2 fine-tuned with box or point prompts. And as I know, MedSAM3, a Promptable Medical Segmentation Model, is also available now. Other possible baselines include SAM-Adapter; MedCLIP-SAMv2; MedCLIPSeg; CLIP-Guided SAM (Jalilian & Bais, 2026). In addition, most baselines in Table 1 lack medical foundation pretraining (U-Net, nnU-Net, LViT, etc.), so the gains conflate pretraining with method design. Some retrained baselines look undertuned. For example, UNet++ reaches 65.75 Dice on QaTa versus 77.03 for U-Net.
  5. The efficiency claim is framed narrowly. "Params" counts only trainable weights, but inference runs two foundation backbones plus E_c, at 114.88G FLOPs. Only reporting this counts can be confusing at the first glance. The paper should also report total parameters, latency and memory against the from-scratch baselines.

Additional weaknesses: (1) Results come from a single seed (42) with no confidence intervals or significance tests. Several "wins" are within noise: QaTa +0.18 Dice, ISIC +0.29, and HD95 on QaTa is not the best. (2) Validation splits exist, but the ablation, placement, schedule and LR/batch tables report the same numbers as the main test results. This suggests design and hyperparameter selection on the test set. The selection protocol must be clarified. (3) Possible pretraining overlap and limited scope. MedSAM and MedSAM2 training corpora and BiomedCLIP's PMC-15M may overlap with Kvasir-SEG, ISIC, or the COVID CXR/CT sources . This should be checked and disclosed.

Questions for authors
  1. How does a parameter-matched MedSAM2 encoder LoRA or adapter (no source features) compare with HCST?
  2. Were the layers, schedules, LR and batch size chosen on the validation or the test split?
  3. What are mean ± std over multiple seeds, and significance against HiMix, DIT and LAVT-RS?
  4. What are total parameters, latency and memory at inference?
  5. Do the pretraining corpora overlap with the evaluation datasets?
Reviewer_mF7wOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes S3-Adapt, a parameter-efficient framework for vision-language-guided medical image segmentation that connects BiomedCLIP with MedSAM2. The framework comprises three components: Reciprocal Semantic Grounding (RSG) constructs text-conditioned spatial representations, Hierarchical Cross-Foundation Semantic Transfer (HCST) injects these representations into multiple stages of the segmentation encoder through gated residual updates, and Evidence-Conditioned Selective Correction (ECSC) refines a fixed reference prediction using local image evidence and prediction context. Both pretrained encoder backbones remain frozen, with 1.97M task-adapted parameters trained through a two-stage procedure. Experiments on four public medical image segmentation datasets report improvements in segmentation accuracy and boundary quality over the evaluated baselines.

Critical strengths

The paper addresses an important limitation of existing vision-language-guided medical image segmentation methods, where semantic information is often incorporated primarily through decoder prompts. By introducing Reciprocal Semantic Grounding (RSG) and Hierarchical Cross-Foundation Semantic Transfer (HCST), the proposed framework enables grounded semantic representations from a vision-language model to interact with intermediate features of a segmentation foundation model. The integration of semantic grounding, hierarchical feature adaptation, and selective prediction refinement forms a coherent architecture with a clear technical motivation.The experimental results demonstrate competitive performance across multiple medical imaging modalities with a relatively small number of trainable parameters.

Critical weaknesses
  1. The paper does not clearly distinguish HCST from existing multimodal feature injection and gated adapter approaches. Moreover, important baselines are missing, including parameter-efficient adaptation of the same MedSAM2 backbone (e.g., LoRA/Adapter) and closely related methods such as MedCLIP-SAMv2, EVF-SAM, and CLIP-Guided SAM. More direct comparisons are needed to substantiate the novelty and effectiveness of the proposed framework.
  2. The experiments are restricted to a single BiomedCLIP–MedSAM2 backbone combination, leaving the transferability of the proposed lightweight modules to other foundation models unexplored. Additionally, although the method requires few trainable parameters, its reliance on two foundation models introduces considerable computational overhead. Reporting inference latency and GPU memory consumption would provide a more comprehensive assessment of its practical efficiency.
  3. It remains unclear whether the improvements achieved by ECSC arise from the proposed evidence-conditioning mechanism or simply from the additional trainable convolutional network. A parameter-matched refinement baseline without evidence conditioning is needed to isolate its contribution.
  4. Potential limitations of reciprocal normalization. The spatial competition introduced by reciprocal normalization may suppress valid region–token associations when a single textual concept corresponds to multiple disconnected regions, such as scattered infection areas. The authors should investigate whether this mechanism negatively affects the recall of multi-region targets.
  5. Insufficient justification for hierarchical injection locations. Although different injection configurations are evaluated, the selection criteria for the final encoder blocks remain unclear. The authors should clarify whether these locations were determined through dataset-specific hyperparameter tuning or based on general architectural considerations.
  6. The image-specific text prompts contain spatial and quantitative attributes, such as target location, number, and size, some of which may be derived from segmentation annotations. If spatial augmentations such as rotation or flipping are applied without updating the corresponding text descriptions, inconsistencies between visual and textual inputs may arise.
Questions for authors

1.What distinguishes HCST from existing multimodal feature injection and gated adapter methods?

2.Are the improvements from ECSC mainly due to evidence conditioning or the additional trainable convolutional network?

3.Could reciprocal normalization suppress valid regions when a text token corresponds to multiple disconnected targets?

4.How were the RSG and HCST injection layers selected? Were they tuned separately for each dataset?

5.Are the test-time text prompts generated independently of ground-truth masks? How is image–text consistency maintained under spatial data augmentation?

6.Can the proposed modules generalize to other backbone combinations? What are the inference latency and GPU memory requirements?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.