- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes BiReS, a text-guided medical image segmentation framework built on LViT. It introduces a Semantic Recalibrator (SR) that uses text embeddings to modulate visual features through spatial gating and FiLM-style channel modulation, and updates the text embedding with attention-pooled visual features. The decoder adds a Multi-Scale Feature Enhancement (MSFE) module with multi-dilation branches and a Tri-Attention (TA) module with pixel, channel, and spatial attention, together with multi-level deep supervision. Experiments on QaTa-COV19, MosMedData+, and BUSI show improvements over vision-only and vision-language baselines in Dice, IoU, HD95, and ASD.
Critical strengths
- BiReS achieves consistent improvements in both region overlap (Dice/IoU) and boundary accuracy (HD95/ASD) on all three datasets.
- The ablation studies are fairly complete, covering each module, SR placement and depth, the vision-to-text update, and deep-supervision weights.
Critical weaknesses
- It is not novel to introduce language-conditioned FiLM modulation, bidirectional cross-modal attention, ASPP-like multi-scale branches, or CBAM-style attention into this pipeline; all have been used in referring and medical segmentation, such as [A-D]. The method reads more like a combination of existing modules. Please discuss and compare with them.
- The inference text already encodes lesion count and location, which is unavailable in practice, so results are closer to an oracle setting. The “w/o text” row is a separately trained model. Please evaluate the same trained BiReS with empty or wrong-location text at test time.
- Several strong baselines cited in the paper are not compared, including the de facto standard nnU-Net and closely related vision-language methods such as [E-G]. Please add these comparisons.
[A] INSIDE: Steering spatial attention with non-imaging information in CNNs. MICCAI, 2020
[B] CE-Net: Context encoder network for 2D medical image segmentation. IEEE Transactions on Medical Imaging, 38(10):2281–2292, 2019.
[C] Recalibrating fully convolutional networks with spatial and channel “squeeze and excitation” blocks. IEEE Transactions on Medical Imaging, 38(2):540–549, 2019.
[D] Medical image segmentation via cascaded attention decoding (CASCADE). WACV, 2023.
[E] Frequency-domain multi-modal fusion for language-guided medical image segmentation (FMISeg). MICCAI, 2025.
[F] Hybrid cross-modality fusion network for medical image segmentation with contrastive learning (HCFNet). Engineering Applications of Artificial Intelligence, 144:110073, 2025.
[G] MIT-SAM: Medical image-text SAM with mutually enhanced heterogeneous feature fusion for medical image segmentation. IEEE Journal of Biomedical and Health Informatics, 2025.
Questions for authors
- MedCLIPSeg’s BUSI split includes normal images, which are removed here. How was the BUSI split rebuilt, and are the results still comparable to those reported in the MedCLIPSeg paper?
- The paper claims to handle large-scale variation, but no results are given by lesion size, and removing MSFE only reduces Dice by 0.33 (Table 3 (a)). Can the authors report results on small, medium, and large lesions?
- Can the authors provide the main ablations (Tables 3 and 5) on MosMedData+?
6 comments
Comment to Reviewer z8M6 (Part 2)
Table R2. Paired slice-level differences between Full SR and each ablated variant on MosMedData+, with 95% bootstrap CIs and Holm-adjusted Wilcoxon p-values.
| Metric | Δ vs. T2V-only [95% CI] | Holm p | Δ vs. w/o SR [95% CI] | Holm p |
|---|---|---|---|---|
| Dice (pp) | +3.86 [3.16, 4.59] | 2.57 × 10⁻²⁵ | +2.20 [1.57, 2.91] | 8.86 × 10⁻¹³ |
| IoU (pp) | +4.14 [3.44, 4.86] | 1.30 × 10⁻²⁴ | +2.41 [1.78, 3.10] | 3.72 × 10⁻¹³ |
| HD95 (px) | −2.02 [−3.50, −0.58] | 0.00915 | −1.86 [−3.37, −0.36] | 0.00915 |
| ASD (px) | −1.12 [−1.66, −0.60] | 8.84 × 10⁻⁶ | −0.76 [−1.18, −0.37] | 5.55 × 10⁻⁶ |
In summary, BiReS contributes an explicit image-conditioned iterative semantic recalibration mechanism and a hierarchical scale-aware feature refinement decoder. While building upon established modulation, convolution, and attention operations, the framework emphasizes progressive semantic state adaptation and hierarchical visual refinement to address cross-modal granularity mismatch and lesion scale heterogeneity.
[W2] Robustness to Missing and Incorrect Location Information in Textual Prompts
We agree that supplying the annotated lesion count and location at inference makes the original evaluation an oracle-prompt setting. The “w/o text” row answers a different question because it uses a separately trained model. To measure the effect of the inference prompt directly, we evaluated one fixed BiReS checkpoint per dataset on the same test images. We supplied the original annotation, an empty string encoded by the same text encoder, or a prompt in which we changed only the location (e.g., “lower right lung” to “lower left lung”), preserving lesion count and all other descriptors.
The tables report per-image means. Both distances are measured in pixels on the 224 × 224 evaluation grid. Every image in each reported condition had finite values for all four metrics. We evaluated empty prompts on every test image. For wrong-location prompts, we compared all three conditions on the same eligible images. We excluded 677 QaTa-COV19 images and 6 MosMedData+ slices from that comparison because the location edits preserved the annotated distribution, as when both lungs were affected throughout.
Table R5. Segmentation overlap under original, empty, and incorrect-location textual prompts. Each entry reports Dice / IoU (%).
| Dataset | Test images used | Original prompt | Empty prompt | Wrong-location prompt |
|---|---|---|---|---|
| QaTa-COV19 | Full set (2,113) | 85.34 / 77.14 | 64.76 / 53.80 | — |
| QaTa-COV19 | Location subset (1,436) | 80.36 / 69.97 | 52.55 / 39.77 | 62.36 / 52.06 |
| MosMedData+ | Full set (273) | 76.10 / 62.96 | 73.44 / 59.88 | — |
| MosMedData+ | Location subset (267) | 75.89 / 62.69 | 73.23 / 59.64 | 75.77 / 62.58 |
| BUSI | Full set (all 65 eligible) | 83.46 / 75.58 | 47.91 / 36.24 | 76.26 / 68.09 |
Table R6. Boundary accuracy under original, empty, and incorrect-location textual prompts. Each entry reports HD95 / ASD (pixels).
| Dataset | Test images used | Original prompt | Empty prompt | Wrong-location prompt |
|---|---|---|---|---|
| QaTa-COV19 | Full set (2,113) | 14.79 / 3.61 | 51.51 / 18.48 | — |
| QaTa-COV19 | Location subset (1,436) | 19.29 / 4.73 | 68.01 / 26.10 | 53.32 / 22.54 |
| MosMedData+ | Full set (273) | 18.87 / 3.36 | 27.52 / 6.60 | — |
| MosMedData+ | Location subset (267) | 19.06 / 3.39 | 27.73 / 6.71 | 18.14 / 3.57 |
| BUSI | Full set (all 65 eligible) | 18.97 / 7.67 | 72.30 / 29.21 | 33.70 / 15.20 |
Removing the prompt reduced full-set Dice by 20.58 percentage points on QaTa-COV19, 2.65 on MosMedData+, and 35.55 on BUSI and increased both boundary distances on all three datasets. With lesion count and other descriptors held fixed, changing location alone reduced Dice by 18.00, 0.12, and 7.20 points, respectively, on the matched images. Both boundary distances also increased on QaTa-COV19 and BUSI. On MosMedData+, HD95 decreased from 19.06 to 18.14 pixels, while ASD increased from 3.39 to 3.57 pixels. Performance with descriptions obtained independently in clinical practice remains to be established.
Comment to Reviewer z8M6 (Part 3)
[W3] Additional Comparisons with Strong Baselines
We agree that comparisons with nnU-Net and recent vision-language segmentation methods, including FMISeg, HCFNet, and MIT-SAM, would provide a more comprehensive evaluation of BiReS. However, due to limited computational resources and the tight rebuttal timeline, we were unable to complete these additional experiments. Moreover, some of the referenced methods lack publicly available PyTorch implementations or have incomplete released code, making direct and reliable reproduction challenging within the available time.
We acknowledge this limitation and plan to reproduce these methods by implementing the necessary components following their original papers, while ensuring consistent dataset splits, training configurations, and evaluation protocols for fair comparisons. We will prioritize these additional comparisons in our subsequent experiments and include the results in a revised manuscript, ensuring a more comprehensive and fair evaluation of BiReS.
[Q1] Clarification of the BUSI Dataset Split and Comparison Fairness
We followed the original BUSI dataset split protocol provided by MedCLIPSeg, without performing any additional random splitting. Specifically, we excluded normal images from the predefined training, validation, and test subsets, respectively, as our study focuses on lesion segmentation. The remaining benign and malignant samples retain their original split assignments.
Importantly, all methods compared in our paper, including MedCLIPSeg, were reproduced and evaluated under the same lesion-only BUSI protocol, ensuring consistent training and evaluation conditions. Therefore, our reported MedCLIPSeg results are directly comparable to BiReS under this shared protocol, but should not be directly compared with the original MedCLIPSeg paper's results obtained using normal images. We will further clarify this distinction and provide detailed dataset statistics in a subsequent revised version of the manuscript.
Comment to Reviewer z8M6 (Part 1)
[W1] Methodological Novelty and Distinctions from Existing Works [A–G]
Thank you for highlighting these relevant studies. We acknowledge that FiLM-style modulation, multi-scale convolution, and spatial-channel attention have been explored in previous research. However, BiReS does not claim novelty in these individual operators. Instead, our contributions lie in image-conditioned iterative semantic recalibration and a specifically designed hierarchical scale-aware feature refinement decoder. We clarify the distinctions below.
(1) Distinctions from existing feature modulation and recalibration methods [A, C]. INSIDE [A] introduces spatially dependent feature modulation conditioned on non-imaging information, while scSE [C] recalibrates visual features through spatial and channel-wise excitation. In contrast, our SR performs mutual recalibration of both visual and textual representations rather than solely modulating visual features. Specifically, global textual semantics first recalibrate deep visual features through spatial gating and FiLM-style channel modulation. The recalibrated visual features then provide image-specific evidence to update the global semantic state through semantic-query-guided attention pooling and residual updating (Eqs. (1)–(2)). Crucially, the updated semantic state and visual features are propagated to subsequent SR layers, enabling iterative semantic recalibration. This allows subsequent visual modulation to depend on progressively image-adapted semantics rather than repeatedly using the initial text representation.
(2) Distinctions from existing multi-scale and attention-based decoders [B–D]. CE-Net [B] employs dense atrous convolution and residual multi-kernel pooling for contextual feature extraction, while CASCADE [D] leverages cascaded attention decoding to integrate hierarchical visual features. We acknowledge that these methods share certain design principles with ours. However, we design a hierarchical scale-aware feature refinement decoder that combines multi-scale contextual aggregation with sequential feature refinement. Specifically, MSFE integrates local, multi-scale, and global contextual information after hierarchical feature fusion, while TA further refines the aggregated representations through pixel, channel, and spatial attention. Unlike scSE [C], which focuses on spatial and channel excitation, TA additionally employs pixel-level attention with learned multi-statistic descriptors. Our contribution is therefore the task-oriented decoder architecture rather than the novelty of individual convolution or attention operations.
(3) Relationship to recent vision-language methods [E–G]. FMISeg [E] emphasizes language-guided frequency-domain feature interaction, HCFNet [F] employs contrastive learning and local/non-local cross-modal fusion, and MIT-SAM [G] focuses on interactive heterogeneous feature fusion and text reconstruction. In contrast, BiReS explicitly maintains and iteratively updates a persistent global semantic state using image-specific visual evidence, allowing the updated semantic representation to guide subsequent visual recalibration. We will further clarify these methodological differences in the revised manuscript and provide additional comparisons under consistent evaluation protocols where feasible.
(4) Experimental evidence. The effectiveness of SR is supported by ablations on QaTa-COV19 and MosMedData+. On QaTa-COV19, Full SR improves Dice by 0.46 percentage points over T2V-only (Table 5, original submission), while removing MSFE and TA reduces Dice by 0.33 and 0.87 points, respectively (Table 3(a), original submission). On MosMedData+, experiments with three matched seeds show that Full SR achieves 75.23 ± 0.87% Dice, outperforming T2V-only and w/o SR by 3.86 and 2.20 percentage points, respectively (Table R1). Paired slice-level tests with Holm correction further support these Dice improvements (both adjusted p < 0.001; Table R2), although run-level significance remains inconclusive with only three seeds.
Table R1. Segmentation performance of SR variants on MosMedData+ (mean ± SD across three training seeds).