acceptodds
Under review as a conference paper at ICLR 2027

BiReS: Bidirectional Semantic Recalibration for Text-Guided Multi-Scale Medical Image Segmentation

10 views

Abstract

Medical image segmentation remains challenging due to substantial variations in target scale, complex and poorly defined boundaries, and the limited semantic awareness of vision-only models. Although multimodal approaches introduce complementary semantic priors, the granularity gap between global textual semantics and dense visual representations complicates effective cross-modal fusion. Meanwhile, many existing methods treat language primarily as auxiliary guidance for visual features, with limited image-conditioned feedback to textual representations, resulting in asymmetric cross-modal interaction. To address these limitations, we propose **BiReS**, a multimodal segmentation framework that combines bidirectional semantic recalibration with scale-aware feature refinement. Specifically, we introduce a **Semantic Recalibrator (SR)** that leverages global textual semantics to modulate visual responses along both spatial and channel dimensions, while attention-weighted visual pooling reciprocally updates textual representations, forming a closed-loop semantic recalibration process. To handle target scale heterogeneity and hierarchical feature mismatch, we introduce a **Multi-Scale Feature Enhancement (MSFE)** module in each decoder stage to refine fused hierarchical features with local, multi-scale, and global context. A **Tri-Attention (TA)** mechanism further refines decoder features, suppressing irrelevant responses while enhancing salient structures and fine-grained boundary cues. Extensive experiments on multiple medical image segmentation benchmarks demonstrate that BiReS consistently outperforms state-of-the-art vision-only and vision-language methods, with notable gains on challenging targets exhibiting substantial scale variation and complex boundary structures.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Medical Image Segmentation, Vision-Language Segmentation, Multimodal Learning, Cross-Modal Interaction, Bidirectional Semantic Recalibration, Multi-Scale Representation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_z8M6OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes BiReS, a text-guided medical image segmentation framework built on LViT. It introduces a Semantic Recalibrator (SR) that uses text embeddings to modulate visual features through spatial gating and FiLM-style channel modulation, and updates the text embedding with attention-pooled visual features. The decoder adds a Multi-Scale Feature Enhancement (MSFE) module with multi-dilation branches and a Tri-Attention (TA) module with pixel, channel, and spatial attention, together with multi-level deep supervision. Experiments on QaTa-COV19, MosMedData+, and BUSI show improvements over vision-only and vision-language baselines in Dice, IoU, HD95, and ASD.

Critical strengths
  1. BiReS achieves consistent improvements in both region overlap (Dice/IoU) and boundary accuracy (HD95/ASD) on all three datasets.
  2. The ablation studies are fairly complete, covering each module, SR placement and depth, the vision-to-text update, and deep-supervision weights.
Critical weaknesses
  1. It is not novel to introduce language-conditioned FiLM modulation, bidirectional cross-modal attention, ASPP-like multi-scale branches, or CBAM-style attention into this pipeline; all have been used in referring and medical segmentation, such as [A-D]. The method reads more like a combination of existing modules. Please discuss and compare with them.
  2. The inference text already encodes lesion count and location, which is unavailable in practice, so results are closer to an oracle setting. The “w/o text” row is a separately trained model. Please evaluate the same trained BiReS with empty or wrong-location text at test time.
  3. Several strong baselines cited in the paper are not compared, including the de facto standard nnU-Net and closely related vision-language methods such as [E-G]. Please add these comparisons.

[A] INSIDE: Steering spatial attention with non-imaging information in CNNs. MICCAI, 2020

[B] CE-Net: Context encoder network for 2D medical image segmentation. IEEE Transactions on Medical Imaging, 38(10):2281–2292, 2019.

[C] Recalibrating fully convolutional networks with spatial and channel “squeeze and excitation” blocks. IEEE Transactions on Medical Imaging, 38(2):540–549, 2019.

[D] Medical image segmentation via cascaded attention decoding (CASCADE). WACV, 2023.

[E] Frequency-domain multi-modal fusion for language-guided medical image segmentation (FMISeg). MICCAI, 2025.

[F] Hybrid cross-modality fusion network for medical image segmentation with contrastive learning (HCFNet). Engineering Applications of Artificial Intelligence, 144:110073, 2025.

[G] MIT-SAM: Medical image-text SAM with mutually enhanced heterogeneous feature fusion for medical image segmentation. IEEE Journal of Biomedical and Health Informatics, 2025.

Questions for authors
  1. MedCLIPSeg’s BUSI split includes normal images, which are removed here. How was the BUSI split rebuilt, and are the results still comparable to those reported in the MedCLIPSeg paper?
  2. The paper claims to handle large-scale variation, but no results are given by lesion size, and removing MSFE only reduces Dice by 0.33 (Table 3 (a)). Can the authors report results on small, medium, and large lesions?
  3. Can the authors provide the main ablations (Tables 3 and 5) on MosMedData+?
6 comments
AuthorsOpenReview ↗

Comment to Reviewer z8M6 (Part 1)

[W1] Methodological Novelty and Distinctions from Existing Works [A–G]

Thank you for highlighting these relevant studies. We acknowledge that FiLM-style modulation, multi-scale convolution, and spatial-channel attention have been explored in previous research. However, BiReS does not claim novelty in these individual operators. Instead, our contributions lie in image-conditioned iterative semantic recalibration and a specifically designed hierarchical scale-aware feature refinement decoder. We clarify the distinctions below.

(1) Distinctions from existing feature modulation and recalibration methods [A, C]. INSIDE [A] introduces spatially dependent feature modulation conditioned on non-imaging information, while scSE [C] recalibrates visual features through spatial and channel-wise excitation. In contrast, our SR performs mutual recalibration of both visual and textual representations rather than solely modulating visual features. Specifically, global textual semantics first recalibrate deep visual features through spatial gating and FiLM-style channel modulation. The recalibrated visual features then provide image-specific evidence to update the global semantic state through semantic-query-guided attention pooling and residual updating (Eqs. (1)–(2)). Crucially, the updated semantic state and visual features are propagated to subsequent SR layers, enabling iterative semantic recalibration. This allows subsequent visual modulation to depend on progressively image-adapted semantics rather than repeatedly using the initial text representation.

(2) Distinctions from existing multi-scale and attention-based decoders [B–D]. CE-Net [B] employs dense atrous convolution and residual multi-kernel pooling for contextual feature extraction, while CASCADE [D] leverages cascaded attention decoding to integrate hierarchical visual features. We acknowledge that these methods share certain design principles with ours. However, we design a hierarchical scale-aware feature refinement decoder that combines multi-scale contextual aggregation with sequential feature refinement. Specifically, MSFE integrates local, multi-scale, and global contextual information after hierarchical feature fusion, while TA further refines the aggregated representations through pixel, channel, and spatial attention. Unlike scSE [C], which focuses on spatial and channel excitation, TA additionally employs pixel-level attention with learned multi-statistic descriptors. Our contribution is therefore the task-oriented decoder architecture rather than the novelty of individual convolution or attention operations.

(3) Relationship to recent vision-language methods [E–G]. FMISeg [E] emphasizes language-guided frequency-domain feature interaction, HCFNet [F] employs contrastive learning and local/non-local cross-modal fusion, and MIT-SAM [G] focuses on interactive heterogeneous feature fusion and text reconstruction. In contrast, BiReS explicitly maintains and iteratively updates a persistent global semantic state using image-specific visual evidence, allowing the updated semantic representation to guide subsequent visual recalibration. We will further clarify these methodological differences in the revised manuscript and provide additional comparisons under consistent evaluation protocols where feasible.

(4) Experimental evidence. The effectiveness of SR is supported by ablations on QaTa-COV19 and MosMedData+. On QaTa-COV19, Full SR improves Dice by 0.46 percentage points over T2V-only (Table 5, original submission), while removing MSFE and TA reduces Dice by 0.33 and 0.87 points, respectively (Table 3(a), original submission). On MosMedData+, experiments with three matched seeds show that Full SR achieves 75.23 ± 0.87% Dice, outperforming T2V-only and w/o SR by 3.86 and 2.20 percentage points, respectively (Table R1). Paired slice-level tests with Holm correction further support these Dice improvements (both adjusted p < 0.001; Table R2), although run-level significance remains inconclusive with only three seeds.

Table R1. Segmentation performance of SR variants on MosMedData+ (mean ± SD across three training seeds).

VariantDice (%) ↑IoU (%) ↑HD95 (px) ↓ASD (px) ↓
Full SR75.23 ± 0.8761.96 ± 0.9920.06 ± 1.443.87 ± 0.54
T2V-only71.37 ± 4.5057.81 ± 4.7422.10 ± 4.865.00 ± 2.46
w/o SR73.03 ± 1.2059.54 ± 1.2521.99 ± 0.574.67 ± 0.44
AuthorsOpenReview ↗

Comment to Reviewer z8M6 (Part 2)

Table R2. Paired slice-level differences between Full SR and each ablated variant on MosMedData+, with 95% bootstrap CIs and Holm-adjusted Wilcoxon p-values.

MetricΔ vs. T2V-only [95% CI]Holm pΔ vs. w/o SR [95% CI]Holm p
Dice (pp)+3.86 [3.16, 4.59]2.57 × 10⁻²⁵+2.20 [1.57, 2.91]8.86 × 10⁻¹³
IoU (pp)+4.14 [3.44, 4.86]1.30 × 10⁻²⁴+2.41 [1.78, 3.10]3.72 × 10⁻¹³
HD95 (px)−2.02 [−3.50, −0.58]0.00915−1.86 [−3.37, −0.36]0.00915
ASD (px)−1.12 [−1.66, −0.60]8.84 × 10⁻⁶−0.76 [−1.18, −0.37]5.55 × 10⁻⁶

In summary, BiReS contributes an explicit image-conditioned iterative semantic recalibration mechanism and a hierarchical scale-aware feature refinement decoder. While building upon established modulation, convolution, and attention operations, the framework emphasizes progressive semantic state adaptation and hierarchical visual refinement to address cross-modal granularity mismatch and lesion scale heterogeneity.

[W2] Robustness to Missing and Incorrect Location Information in Textual Prompts

We agree that supplying the annotated lesion count and location at inference makes the original evaluation an oracle-prompt setting. The “w/o text” row answers a different question because it uses a separately trained model. To measure the effect of the inference prompt directly, we evaluated one fixed BiReS checkpoint per dataset on the same test images. We supplied the original annotation, an empty string encoded by the same text encoder, or a prompt in which we changed only the location (e.g., “lower right lung” to “lower left lung”), preserving lesion count and all other descriptors.

The tables report per-image means. Both distances are measured in pixels on the 224 × 224 evaluation grid. Every image in each reported condition had finite values for all four metrics. We evaluated empty prompts on every test image. For wrong-location prompts, we compared all three conditions on the same eligible images. We excluded 677 QaTa-COV19 images and 6 MosMedData+ slices from that comparison because the location edits preserved the annotated distribution, as when both lungs were affected throughout.

Table R5. Segmentation overlap under original, empty, and incorrect-location textual prompts. Each entry reports Dice / IoU (%).

DatasetTest images usedOriginal promptEmpty promptWrong-location prompt
QaTa-COV19Full set (2,113)85.34 / 77.1464.76 / 53.80—
QaTa-COV19Location subset (1,436)80.36 / 69.9752.55 / 39.7762.36 / 52.06
MosMedData+Full set (273)76.10 / 62.9673.44 / 59.88—
MosMedData+Location subset (267)75.89 / 62.6973.23 / 59.6475.77 / 62.58
BUSIFull set (all 65 eligible)83.46 / 75.5847.91 / 36.2476.26 / 68.09

Table R6. Boundary accuracy under original, empty, and incorrect-location textual prompts. Each entry reports HD95 / ASD (pixels).

DatasetTest images usedOriginal promptEmpty promptWrong-location prompt
QaTa-COV19Full set (2,113)14.79 / 3.6151.51 / 18.48—
QaTa-COV19Location subset (1,436)19.29 / 4.7368.01 / 26.1053.32 / 22.54
MosMedData+Full set (273)18.87 / 3.3627.52 / 6.60—
MosMedData+Location subset (267)19.06 / 3.3927.73 / 6.7118.14 / 3.57
BUSIFull set (all 65 eligible)18.97 / 7.6772.30 / 29.2133.70 / 15.20

Removing the prompt reduced full-set Dice by 20.58 percentage points on QaTa-COV19, 2.65 on MosMedData+, and 35.55 on BUSI and increased both boundary distances on all three datasets. With lesion count and other descriptors held fixed, changing location alone reduced Dice by 18.00, 0.12, and 7.20 points, respectively, on the matched images. Both boundary distances also increased on QaTa-COV19 and BUSI. On MosMedData+, HD95 decreased from 19.06 to 18.14 pixels, while ASD increased from 3.39 to 3.57 pixels. Performance with descriptions obtained independently in clinical practice remains to be established.

AuthorsOpenReview ↗

Comment to Reviewer z8M6 (Part 3)

[W3] Additional Comparisons with Strong Baselines

We agree that comparisons with nnU-Net and recent vision-language segmentation methods, including FMISeg, HCFNet, and MIT-SAM, would provide a more comprehensive evaluation of BiReS. However, due to limited computational resources and the tight rebuttal timeline, we were unable to complete these additional experiments. Moreover, some of the referenced methods lack publicly available PyTorch implementations or have incomplete released code, making direct and reliable reproduction challenging within the available time.

We acknowledge this limitation and plan to reproduce these methods by implementing the necessary components following their original papers, while ensuring consistent dataset splits, training configurations, and evaluation protocols for fair comparisons. We will prioritize these additional comparisons in our subsequent experiments and include the results in a revised manuscript, ensuring a more comprehensive and fair evaluation of BiReS.

[Q1] Clarification of the BUSI Dataset Split and Comparison Fairness

We followed the original BUSI dataset split protocol provided by MedCLIPSeg, without performing any additional random splitting. Specifically, we excluded normal images from the predefined training, validation, and test subsets, respectively, as our study focuses on lesion segmentation. The remaining benign and malignant samples retain their original split assignments.

Importantly, all methods compared in our paper, including MedCLIPSeg, were reproduced and evaluated under the same lesion-only BUSI protocol, ensuring consistent training and evaluation conditions. Therefore, our reported MedCLIPSeg results are directly comparable to BiReS under this shared protocol, but should not be directly compared with the original MedCLIPSeg paper's results obtained using normal images. We will further clarify this distinction and provide detailed dataset statistics in a subsequent revised version of the manuscript.

Reviewer_4x4hOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on text-guided medical image segmentation. To address challenges such as the discrepancy between global textual semantics and spatial features, the unidirectional nature of text-based guidance for visual features, and the variations across multi-scale features, the authors propose BiReS. In this framework, the SR module handles cross-modal interaction, while the MSFE and TA modules facilitate multi-scale and attention-driven enhancement within the decoder. The proposed method has been experimentally validated on three 2D medical image segmentation datasets.

Critical strengths
  1. There is a granularity difference between text semantics and pixel-level features, which is also a difficulty in medical image and text segmentation. The research questions have practical significance and the motivation for the architecture is clear.
  2. The model has a good degree of modularity. SR mainly handles cross-modal interaction, MSFE is responsible for multi-scale context fusion, TA is responsible for attention enhancement, and deep supervision acts on the optimization process. Each part has a certain division of labor in terms of design motivation.
Critical weaknesses
  1. The paper appears to be a combinatorial optimization of several existing works. Specifically, the bidirectional update operation in the "SR" module has already been explored in various studies, such as C2Seg (ICLR 2026) and MedCLIPSeg (CVPR 2026). The MSFE module appears to adopt an ASPP-like multi-branch design. The three attention mechanisms within "TA" are also well-established techniques. Furthermore, the performance gain from the deep supervision weighting strategy is marginal and hardly qualifies as a significant innovation.
  2. The boundaries between the "MSFE" and "TA" modules are somewhat ambiguous; while it appears that "TA" is a component of "MSFE," Table 3(a) treats them as independently removable modules. Although the paper emphasizes the lightweight nature of "TA," it fails to provide metrics such as FLOPs or inference latency for the complete model.
Questions for authors
  1. What are the independent innovations of this paper compared to existing work? Please clearly explain the methodological differences and the contributions of the paper, specifically the substantive distinction between your proposed SR module and existing bidirectional cross-modal interaction mechanisms.
  2. What are the specific implementation boundaries for the "w/o MSFE" and "w/o TA" variants in Table 3? Additionally, could you provide further details, such as the FLOPs and inference latency for each method?
3 comments
AuthorsOpenReview ↗

Comment to Reviewer 4x4h (Part 1)

[W1, Q1] Methodological Novelty and Independent Contributions

Thank you for the constructive comments regarding the methodological novelty of BiReS. We acknowledge that bidirectional vision-language interaction, multi-scale convolution, and attention mechanisms have been investigated in previous studies. However, our contributions lie in the image-conditioned iterative semantic recalibration mechanism and a specifically designed hierarchical scale-aware feature refinement decoder, rather than the novelty of individual operators. We clarify the distinctions below.

(1) Independent development and distinctions from existing methods. We would first like to clarify that the initial conceptualization and architectural design of BiReS were completed independently, before we became aware of C2Seg (ICLR 2026) and MedCLIPSeg (CVPR 2026). We encountered MedCLIPSeg while identifying relevant baseline methods for experimental comparison and subsequently included it in our comparative experiments. Methodologically, C2Seg addresses text-mask consistency through cluster-aware contrastive learning and bidirectional complementary attention. Its BCAM operates on channel-aligned token-level language features (N×CN \times C) and spatial visual features (P×CP \times C), primarily emphasizing complementary cross-modal interactions and spatial feature fusion. Meanwhile, MedCLIPSeg focuses on probabilistic patch-level cross-modal interaction and uncertainty modeling. In contrast, SR goes beyond cross-modal feature fusion by explicitly recalibrating both visual and textual representations and iteratively updating a persistent global semantic state using image-specific visual evidence. The updated semantic state is subsequently propagated to deeper SR layers to guide further visual recalibration.

(2) Mutual recalibration and iterative semantic state updating. Unlike conventional bidirectional feature fusion, SR explicitly recalibrates both visual and textual representations. As shown in Eqs. (1) and (2), global textual semantics first modulate deep visual features through spatial gating and FiLM-style channel modulation (Text-to-Vision Recalibration). Subsequently, semantic-query-guided attention pooling extracts image-specific evidence from the recalibrated visual features to update the global semantic state (Visual-to-Text Updating). Crucially, both the recalibrated visual features and updated semantic state are propagated to the next SR layer, enabling iterative mutual recalibration. Unlike BRINet's region-adaptive vision-language attention and MIT-SAM's interactive heterogeneous feature fusion, SR explicitly maintains a global semantic state that is progressively adapted using visual evidence to guide subsequent visual recalibration, rather than repeatedly conditioning on the same initial global text embedding.

The initial semantic representation is extracted using a frozen CLIP text encoder. During forward propagation, intermediate SR layers dynamically update the semantic state, while the final SR layer forwards only the recalibrated visual features to the decoder. The intermediate semantic updating branches are optimized end-to-end through their influence on subsequent visual modulation and the segmentation loss. Thus, image-conditioned feedback operates through iterative semantic state adaptation without modifying the frozen text encoder.

(3) Hierarchical scale-aware decoder. Beyond SR, we design a hierarchical scale-aware feature refinement decoder to address lesion scale heterogeneity and semantic discrepancies between hierarchical visual features. Specifically, MSFE aggregates complementary local, multi-scale, and global contextual information, while TA subsequently refines the aggregated representations through pixel, channel, and spatial attention. Although MSFE shares similarities with ASPP-like structures and TA employs established attention operations, our contribution lies in their task-oriented integration into a coherent decoder for hierarchical feature refinement, rather than introducing entirely new convolution or attention operators. We also acknowledge that the improvement from the deep-supervision weighting schedule over uniform weighting is modest (0.11 percentage points in Dice). Therefore, we regard it as an auxiliary optimization strategy rather than an independent methodological innovation.

AuthorsOpenReview ↗

Comment to Reviewer 4x4h (Part 2)

(4) Experimental evidence. The effectiveness of SR is supported by experiments on both QaTa-COV19 and MosMedData+. On QaTa-COV19, Full SR improves Dice by 0.46 percentage points and reduces HD95 by 0.50 pixels over T2V-only (Table 5, original submission), while Fig. 3 illustrates enhanced responses to small lesions and complex boundaries. On MosMedData+, experiments with three training seeds (10, 42, 66) show that Full SR achieves 75.23 ± 0.87% Dice, outperforming T2V-only and w/o SR by 3.86 and 2.20 percentage points, respectively (Table R1). Paired slice-level tests with Holm correction confirm significant Dice improvements (95% CIs: [3.16, 4.59] and [1.57, 2.91]; both adjusted p < 0.001; Table R2). Exploratory group-level tests further support significant Dice and IoU gains over T2V-only (p = 0.03125), although run-level significance remains inconclusive with only three seeds.

Table R1. Segmentation performance of SR variants on MosMedData+ (mean ± SD across three training seeds).

VariantDice (%) ↑IoU (%) ↑HD95 (px) ↓ASD (px) ↓
Full SR75.23 ± 0.8761.96 ± 0.9920.06 ± 1.443.87 ± 0.54
T2V-only71.37 ± 4.5057.81 ± 4.7422.10 ± 4.865.00 ± 2.46
w/o SR73.03 ± 1.2059.54 ± 1.2521.99 ± 0.574.67 ± 0.44

Table R2. Paired slice-level differences between Full SR and each ablated variant on MosMedData+, with 95% bootstrap CIs and Holm-adjusted Wilcoxon p-values.

MetricΔ vs. T2V-only [95% CI]Holm pΔ vs. w/o SR [95% CI]Holm p
Dice (pp)+3.86 [3.16, 4.59]2.57 × 10⁻²⁵+2.20 [1.57, 2.91]8.86 × 10⁻¹³
IoU (pp)+4.14 [3.44, 4.86]1.30 × 10⁻²⁴+2.41 [1.78, 3.10]3.72 × 10⁻¹³
HD95 (px)−2.02 [−3.50, −0.58]0.00915−1.86 [−3.37, −0.36]0.00915
ASD (px)−1.12 [−1.66, −0.60]8.84 × 10⁻⁶−0.76 [−1.18, −0.37]5.55 × 10⁻⁶

In summary, BiReS makes two complementary methodological contributions: (i) image-conditioned iterative semantic recalibration, which explicitly updates and propagates a persistent global semantic state to guide subsequent visual refinement; and (ii) a hierarchical scale-aware decoder designed to enhance multi-scale contextual representations and fine-grained lesion delineation. We will revise the manuscript to clarify these contributions and more precisely distinguish our approach from existing bidirectional interaction and feature enhancement methods.

AuthorsOpenReview ↗

Comment to Reviewer 4x4h (Part 3)

[W2, Q2] Clarification of Ablation Boundaries and Computational Efficiency

(1) Clarification of the ablation configurations.

In our implementation, Tri-Attention (TA) serves as a reusable feature refinement mechanism incorporated into both the Multi-Scale Feature Enhancement (MSFE) modules and the DualScales refinement module. Although TA is integrated into MSFE in the architectural formulation, it can be independently disabled without removing the underlying multi-scale feature extraction and fusion operations.

Specifically, the "w/o TA" variant removes all TA instances throughout the network, including those embedded in MSFE and DualScales, while preserving their remaining feature extraction and fusion operations. Thus, this configuration evaluates the overall contribution of attention-based feature refinement rather than merely removing TA from MSFE.

(2) Computational cost and inference efficiency.

To provide a comprehensive evaluation of computational efficiency, we additionally measured the floating-point operations (FLOPs) and inference latency of BiReS and all compared baselines under consistent experimental settings. The results are summarized in Table R4.

Table R4. Computational complexity and inference latency of BiReS and competing methods

MethodsFLOPs (G)Mean inference latency (ms/image)
UNet73.7562.591
TransUNet58.3369.620
Swin-UMamba64.0739.671
nnWNet16.0966.794
LViT67.5379.253
RecLMIS49.0798.529
Thread22.9739.114
MMI-UNet21.37311.551
DD-CMD23.6779.816
MedCLIPSeg184.75027.965
BiReS w/o text106.05719.060
BiReS (Ours)124.21426.092

All measurements were conducted on an NVIDIA GeForce RTX 4090 GPU using 224 × 224 input images and BF16 mixed-precision inference. To ensure consistent timing measurements, each model was evaluated using 20 warm-up iterations followed by 100 synchronized forward passes on a fixed test batch. The reported latency represents the mean inference time per image, excluding data loading and external preprocessing.

FLOPs were estimated using the convention of 1 MAC = 2 FLOPs. The estimates include supported operations executed within frozen CLIP encoders, where applicable, as well as analytical estimates for selective-scan operations. Most normalization, activation, and interpolation operations were excluded from FLOPs counting. Therefore, the reported FLOPs represent approximate theoretical computational costs rather than exhaustive operation counts.

Reviewer_EMZuOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes BiReS, a text-guided medical image segmentation framework that addresses the granularity mismatch between textual and visual representations, asymmetric cross-modal interaction, and variations in lesion scale. Built on LViT backbone with a frozen CLIP text encoder, the framework comprises three modules:

  1. SR (Semantic Recalibrator) performs bidirectional interaction (text-to-image and image-to-text). Textual semantics guide channel-wise modulation of visual features, while attention-weighted pooling extracts image-specific visual information to update the semantic representation.
  2. MSFE (Multi-Scale Feature Enhancement) combines local details with broader contextual information to handle lesions of varying sizes. The aggregated features are further processed by Tri-Attention to accommodate lesion scale variation.
  3. TA (Tri-Attention) applies pixel, channel, and spatial attention to reweight visual features, and enhance informative structures and boundary-related cues. BiReS achieves Dice/IoU scores of 85.34%/77.14% on QaTa-COV19, 76.10%/62.95% on MosMedData+, and 83.50%/75.62% on BUSI, outperforming all compared image-only and language-guided methods. It also achieves the lowest HD95 and ASD on all three datasets. The authors conduct ablation studies on QaTa-COV19 by removing SR, MSFE, and TA, varying deep-supervision weights, examining SR placement and stack depth, and comparing bidirectional SR with text-to-vision-only recalibration.
Critical strengths
  1. Fusing global textual descriptions with dense visual predictions is an important challenge and is a hot research topic at the moment. I agree with the author that updating the semantic state using image-specific evidence is an encouraging direction to improve the performance.
  2. The evaluation has been conducted on the commonly used language guided segmentation benchmark datasets and the improvements are consistent across different datasets.
Critical weaknesses
  1. The technical novelty seems to be limited: 1) the distinctive contribution beyond established bidirectional fusion and fusion based methods is not sufficiently demonstrated. 2) it is also not clear how the proposed method is different from the existing language guided methods e.g., Bidirectional vision–language interaction already appears in BRINet.
  2. The proposed approach also largely builds on established feature modulation, attention pooling, and multi-scale convolution designs. SR combines spatial gating, FiLM-style channel modulation, and visual feedback for semantic updating, while MSFE and TA further enhance multi-scale features and attention modeling. Given that prior studies already explore these mechanism, the authors have not sufficiently clarified how their specific interaction and semantic updating mechanisms differ from existing approaches, or why these differences are consequential.
  3. The reliability and generality of the central ablation findings are inconsistent. E.g., removing SR reduces Dice by 0.49 percentage points, and full SR improves over text-to-vision-only conditioning by 0.46 points. It’s not clear whether these improvements are statistically significant.
  4. It’s also not clear how the feedback mechanism works e.g., the implementation details of the feedback mechanism are missing.
Questions for authors
  1. Could the authors clarify the substantive differences between SR and prior cross-modal interaction approaches cited in the paper, such as BRINet (Hu et al., 2020) and MIT-SAM (Zhou et al., 2025)? BRINet explicitly models bidirectional vision–language guidance, while MIT-SAM explores interactive image–text fusion for mutual feature enhancement. What distinguishes SR’s semantic updating mechanism from these approaches, and why are these differences consequential?
  2. Reporting performance stratified by lesion size, using the existing test predictions, would help substantiate the specific claim about handling scale heterogeneity.
4 comments
AuthorsOpenReview ↗

Comment to Reviewer EMZu (Part 1)

We sincerely thank you for the constructive comments and thoughtful suggestions regarding the novelty, effectiveness, and statistical reliability of our proposed framework. In response, we have (1) clarified the methodological distinctions between SR and existing bidirectional vision-language interaction approaches; (2) conducted additional ablation experiments on MosMedData+ with multiple random seeds and statistical significance analyses; (3) elaborated on the implementation and optimization of the image-conditioned feedback mechanism; and (4) evaluated segmentation performance across different lesion sizes on all three datasets.

[W1, W2, Q1] Novelty and Effectiveness of Semantic Recalibration

We acknowledge that bidirectional vision-language interaction has been investigated in previous studies. However, the proposed Semantic Recalibrator (SR) differs from existing approaches, particularly BRINet and MIT-SAM, in both its interaction mechanism and the propagation of image-conditioned semantic representations.

(1) Reciprocal recalibration rather than conventional cross-modal feature fusion.

The central motivation of SR is to enable mutual recalibration between global textual semantics and deep visual representations. Global textual embeddings encode high-level semantic information, including lesion categories and spatial descriptions, which can provide complementary guidance for refining visual features. Conversely, visual representations contain rich image-specific contextual information that can be used to adapt and enrich the global semantic representation.

As formulated in Eqs. (1) and (2), SR first performs text-to-vision (T2V) recalibration, where the current global semantic state generates spatial gating weights and FiLM-style channel modulation parameters to adaptively refine visual features. Subsequently, vision-to-text (V2T) updating aggregates image-specific evidence from the recalibrated visual representations through semantic-query-guided attention pooling. The aggregated visual context is then incorporated into the global semantic state through residual updating and layer normalization.

Consequently, an intermediate SR layer produces both recalibrated visual features and an updated semantic state, enabling reciprocal adaptation between the two modalities.

In contrast, BRINet primarily models bidirectional vision-language relationships through cross-modal attention, while MIT-SAM employs interactive image-text fusion to enhance heterogeneous representations. Specifically, MIT-SAM performs image-to-text interaction using Transformer-refined visual features and subsequently employs text-to-image interaction to generate enhanced cross-modal representations. Although these methods also exploit information from both modalities, their primary emphasis is on cross-modal interaction and feature fusion, whereas SR explicitly maintains and updates a global semantic state for subsequent visual recalibration.

(2) Iterative propagation of image-conditioned semantic states across SR layers.

A further distinction lies in how semantic information is propagated through successive interaction layers.

In BiReS, SR layers are stacked to progressively recalibrate deep visual representations and global textual semantics. Importantly, the updated semantic state produced by an intermediate SR layer is explicitly propagated to the subsequent layer, together with the recalibrated visual features. Therefore, the semantic representation used for visual modulation is not fixed throughout the SR stack but is progressively adapted according to image-specific visual evidence.

Specifically, SR employs semantic-query-guided visual evidence aggregation to update a persistent global semantic state, which is subsequently propagated to deeper recalibration layers. This establishes an iterative image-conditioned feedback mechanism rather than treating textual semantics solely as a fixed conditioning signal.

By comparison, BRINet's Bidirectional Cross-modal Attention Module (BCAM) and MIT-SAM's image-text fusion mechanisms primarily focus on establishing cross-modal dependencies and enhancing fused representations. Our distinction is the explicit propagation of an image-recalibrated global semantic state across successive SR layers, which continuously conditions subsequent visual feature modulation.

This design is motivated by the potential discrepancy between the semantic information encoded in a pretrained text embedding and the visual evidence present in a particular medical image. By progressively incorporating image-specific contextual information, SR aims to reduce this discrepancy and provide more adaptive semantic guidance for downstream segmentation.

AuthorsOpenReview ↗

Comment to Reviewer EMZu (Part 2)

Furthermore, BRINet primarily models language information through token-level interactions, whereas SR explicitly operates on a global semantic state and its relationship with dense visual representations. This provides a different mechanism for addressing the granularity mismatch between high-level textual semantics and spatially distributed visual features.

(3) Empirical evidence supporting semantic recalibration.

The qualitative results in Fig. 3 provide additional evidence of the effectiveness of SR. Compared with the variant without SR, incorporating semantic recalibration produces more concentrated responses in small-lesion regions and more complete activations around lesions with complex boundaries.

These observations suggest that image-conditioned semantic recalibration helps the model exploit complementary textual cues, particularly lesion-location information, to improve its sensitivity to clinically relevant regions.

Together with the quantitative ablation results, these findings support the effectiveness of reciprocal semantic recalibration in enhancing visual representations and improving segmentation quality.

[W3] Statistical Significance of the SR Ablation on MosMedData+

Full SR improved mean Dice by 3.86 percentage points over T2V-only and 2.20 points over w/o SR. We computed each metric per test slice and macro-averaged within each run. The table below reports the mean ± sample SD of these macro-averages across three matched training seeds (10, 42, and 66).

Table R1. Segmentation performance of SR variants on MosMedData+ (mean ± SD across three training seeds).

VariantDice (%) ↑IoU (%) ↑HD95 (px) ↓ASD (px) ↓
Full SR75.23 ± 0.8761.96 ± 0.9920.06 ± 1.443.87 ± 0.54
T2V-only71.37 ± 4.5057.81 ± 4.7422.10 ± 4.865.00 ± 2.46
w/o SR73.03 ± 1.2059.54 ± 1.2521.99 ± 0.574.67 ± 0.44

For each slice, we averaged each variant's score across seeds and computed the paired difference (Full SR minus the ablated variant). We report mean paired differences with 95% confidence intervals from 10,000 paired slice bootstrap resamples. We used two-sided Wilcoxon signed-rank tests and applied Holm correction jointly to all eight comparisons. All 273 slices contributed to Dice and IoU, and 272 had finite HD95 and ASD values for all nine models.

Table R2. Paired slice-level differences between Full SR and each ablated variant on MosMedData+, with 95% bootstrap CIs and Holm-adjusted Wilcoxon p-values.

MetricΔ vs. T2V-only [95% CI]Holm pΔ vs. w/o SR [95% CI]Holm p
Dice (pp)+3.86 [3.16, 4.59]2.57 × 10⁻²⁵+2.20 [1.57, 2.91]8.86 × 10⁻¹³
IoU (pp)+4.14 [3.44, 4.86]1.30 × 10⁻²⁴+2.41 [1.78, 3.10]3.72 × 10⁻¹³
HD95 (px)−2.02 [−3.50, −0.58]0.00915−1.86 [−3.37, −0.36]0.00915
ASD (px)−1.12 [−1.66, −0.60]8.84 × 10⁻⁶−0.76 [−1.18, −0.37]5.55 × 10⁻⁶

All eight comparisons favored Full SR at the slice level (Holm-adjusted p < 0.01 throughout). Because neighboring CT slices may be correlated, we also performed an exploratory sign-flip test on nine groups inferred from filename prefixes. After Holm correction, Full SR retained significant gains in Dice and IoU over T2V-only (both p = 0.03125) and a significant reduction in ASD relative to w/o SR (p = 0.046875).

In a sensitivity analysis restricted to matched seeds 10 and 66, Full SR retained its slice-level Dice and IoU gains over both variants (all four adjusted p = 0.0122). Its mean HD95 and ASD differences versus T2V-only reversed direction, indicating that the boundary-distance result for this comparison was sensitive to the choice of training runs. With only three matched seeds, the run-level tests were inconclusive.

AuthorsOpenReview ↗

Comment to Reviewer EMZu (Part 3)

[W4] Image-conditioned Feedback

The image-conditioned feedback mechanism in BiReS is implemented through the explicit vision-to-text semantic updating pathway within SR, as formulated in Eqs. (1) and (2).

Specifically, we employ a frozen pretrained CLIP text encoder to extract an initial global textual embedding, which serves as the initial global semantic state. This representation is jointly processed with deep visual features extracted by the backbone through stacked SR layers, enabling progressive bidirectional semantic recalibration.

In each intermediate SR layer, the current semantic state first guides visual feature recalibration through spatial gating and FiLM-style channel modulation (Text-to-Vision Recalibration). Subsequently, the semantic state is projected into a visual query vector, which performs attention-weighted pooling over the recalibrated visual features to aggregate image-specific visual evidence.

The resulting visual context is then used to update the global semantic state through a residual MLP and layer normalization (Vision-to-Text Updating).

Both the recalibrated visual features and the updated semantic state are propagated to the subsequent SR layer. The updated semantic state therefore participates in the next stage of visual modulation, establishing an iterative feedback process in which textual semantics guide visual refinement while image-specific evidence reciprocally recalibrates the semantic representation.

At the final SR layer, only the recalibrated visual representation is retained and forwarded to the decoder, while the updated semantic state is discarded. Consequently, the effective contribution of image-conditioned feedback is realized through intermediate SR layers, where updated semantic states influence subsequent visual recalibration and ultimately affect segmentation predictions.

During training, the pretrained CLIP text encoder remains frozen, whereas the trainable parameters within SR that contribute to the final prediction are optimized through end-to-end backpropagation.

In particular, the semantic updating branches of intermediate SR layers receive gradient signals from the downstream segmentation objective because their updated semantic representations condition subsequent visual recalibration operations. Thus, the model learns how to incorporate image-specific visual evidence into the semantic state in a manner beneficial to the segmentation task.

Importantly, semantic state updating and parameter optimization are two distinct processes. The semantic state is dynamically updated during each forward pass through successive SR layers, whereas the parameters governing this updating mechanism are learned through backpropagation.

Therefore, image-conditioned feedback in BiReS does not refer to modifying the parameters of the frozen CLIP text encoder. Instead, it explicitly recalibrates and propagates a global semantic state using visual evidence from the current image, allowing subsequent visual feature modulation to adapt to image-specific semantic information.

AuthorsOpenReview ↗

Comment to Reviewer EMZu (Part 4)

[Q2] Performance Stratified by Lesion Size

We use the Lesion Area Ratio (LAR), defined as the ratio of the lesion area to the total pixels of the image, to measure lesion size. Based on the tertiles of the lesion area ratio on the test set, the test samples are divided into three groups: Small, Medium, and Large. We select TransUNet, DD-CMD, and BiReS from the unimodal and multimodal methods respectively for comparison (Table R3). The results of other methods will be presented as an appendix in the revised manuscript later.

Table R3: Comparison of lesion-size-stratified segmentation performance on QaTa-COV19, MosMedData+, and BUSI.

ModelDatasetLesion SizeSamplesDiceIoUHD95ASD
TransUNetQaTa-COV19Small7040.64230.511153.354318.4486
TransUNetQaTa-COV19Medium7050.84480.749526.04404.5204
TransUNetQaTa-COV19Large7040.93450.884210.48281.6591
TransUNetMosMedData+Small910.64330.497226.80367.8181
TransUNetMosMedData+Medium910.71780.579623.01094.7299
TransUNetMosMedData+Large910.75150.614421.38042.1236
TransUNetBUSISmall220.79450.716117.30468.8562
TransUNetBUSIMedium210.76850.678529.521010.9208
TransUNetBUSILarge220.85010.763923.61338.2114
DD-CMDQaTa-COV19Small7040.70950.584625.35878.2585
DD-CMDQaTa-COV19Medium7050.87700.792116.37542.7215
DD-CMDQaTa-COV19Large7040.95360.91386.73991.2883
DD-CMDMosMedData+Small910.68820.549221.32026.0155
DD-CMDMosMedData+Medium910.69420.556122.37793.9042
DD-CMDMosMedData+Large910.77190.641614.35802.0334
DD-CMDBUSISmall220.79470.713013.41386.9905
DD-CMDBUSIMedium210.77580.691327.09599.1331
DD-CMDBUSILarge220.87160.790322.24327.8996
BiReSQaTa-COV19Small7040.72500.601221.80766.9279
BiReSQaTa-COV19Medium7050.88470.803115.59132.5750
BiReSQaTa-COV19Large7040.95050.90986.96681.3138
BiReSMosMedData+Small910.71800.579422.93705.0515
BiReSMosMedData+Medium910.75620.622719.58153.1489
BiReSMosMedData+Large910.80900.686913.75641.8234
BiReSBUSISmall220.80690.731217.88539.9163
BiReSBUSIMedium210.80490.721220.69296.8518
BiReSBUSILarge220.89100.813518.42256.2528

Note: For DD-CMD on the BUSI-Small subgroup, HD95 and ASD were computed from 21 valid samples out of 22, excluding non-finite values. Bold values indicate the best performance within each dataset and lesion-size subgroup.

The results show that BiReS achieves higher Dice and IoU values than TransUNet across all 9 groups formed by three datasets and three lesion sizes. Compared with DD-CMD, BiReS obtains higher Dice and IoU values in 8 of these groups. Particularly for the small lesion group, the Dice scores of BiReS on QaTa-COV19, MosMedData+ and BUSI are 0.7250, 0.7180 and 0.8069, representing improvements of 8.27, 7.47 and 1.24 percentage points over TransUNet, and 1.55, 2.98 and 1.22 percentage points over DD-CMD, respectively. These results demonstrate that the regional segmentation gains of the model cover small, medium and large lesions, and such gains are also observed in the challenging small lesion group.

Meanwhile, we notice that this advantage does not hold for all metrics. For instance, in the large lesion group of QaTa-COV19, the Dice score of BiReS is 0.9505, which is slightly lower than DD-CMD’s 0.9536. In the small lesion group of BUSI, BiReS also fails to achieve the best HD95 and ASD results. Therefore, these stratified results provide supplementary experimental evidence for the model’s ability to handle lesion scale heterogeneity, and indicate that there remains room for improvement in the boundary localization of some small lesions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.