acceptodds
Withdrawn from ICLR 2027

Layer-Wise Spatial Evidence for Medical Visual Grounding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Medical visual grounding requires identifying the intended lesion or anatomy and delineating its precise extent under limited pixel coverage, ambiguous boundaries, and cross-modal domain shift. Existing segmentation-token methods typically select a Transformer layer based on hidden states before constructing its spatial map. However, an isolated hidden state lacks the spatial context to reveal where its eventual spatial map will respond, resulting in suboptimal layer composition and background contamination. We introduce LaSE, the first medical grounding framework to adopt a map-first principle: it evaluates spatial evidence prior to layer fusion. LaSE comprises two novel components: (1) Layer-wise Token Matching (LTM), which constructs a candidate spatial map at every Transformer layer by matching the generated segmentation state with image tokens, exposing layer-wise spatial responses without pre-selecting a layer; and (2) Medical Evidence-Guided Fusion (MEGF), which pools features from the medical regions indicated by each candidate map in a shared frozen MedSAM2 representation and adaptively predicts case-dependent weights, thereby suppressing candidates that point to spurious anatomies. We further propose Lightweight Mask Refinement (LMR), which refines the decoded mask with a lightweight convolutional pyramid and requires only a single MedSAM2 decoder call. Extensive experiments on a four-source (Source-4), ten-target (OOD-10) medical segmentation benchmark demonstrate the effectiveness of LaSE: it achieves 90.20% Source-4 and 84.14% OOD-10 DSC, surpassing the state-of-the-art MedCLIPSeg by 1.09 and 5.12 points on Source-4 and OOD-10, respectively, while improving the pre-decoder map Soft Dice from 54.55% to 59.92%. Code is available at https://anonymous.4open.science/r/LaSE-22.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Medical Visual Grounding, Foundation Model, Text Image Allignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.80 over 5 reviews, above 27% of the others.
Reviewer_uvTeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how representations from different multimodal Transformer layers should be composed into segmentation prompts for medical visual grounding. The proposed LaSE follows a map-first strategy: Layer-wise Token Matching (LTM) constructs candidate spatial maps at each layer; Medical Evidence-Guided Fusion (MEGF) uses MedSAM2 features pooled from the regions indicated by these maps to adaptively fuse both the maps and their corresponding segmentation-token states; and Lightweight Mask Refinement (LMR) refines the decoded mask. Evaluation covers four source and ten out-of-domain datasets. The paper reports a 5.12-point improvement in average OOD DSC over MedCLIPSeg, alongside component ablations and analyses of pre-decoder spatial maps.

Critical strengths
  1. Well-motivated design. LaSE lets layer fusion consider actual spatial responses, addressing the gap between semantic relevance and localization accuracy.
  2. MEGF uses medical image features from candidate regions to predict input-dependent layer weights. Shared fusion weights keep spatial maps and segmentation-token states aligned.
  3. Experiments cover four source and ten OOD datasets, with gains across all ten targets.
Critical weaknesses
  1. The main ablation does not sufficiently isolate the contribution of spatial-evidence-guided fusion. The layer-first control uses hidden states and REINFORCE to select a single layer, whereas the full model uses regional features for soft multi-layer fusion. This changes the information source, fusion mechanism, and optimization procedure simultaneously. Better pre-decoder maps are informative, but do not distinguish the benefit of regional evidence from that of multi-layer aggregation or more direct optimization. Since map-first fusion is the central methodological contribution, the absence of a hidden-state-only control with the same soft-fusion mechanism, and simple baselines such as uniform fusion, limits the strength of the attribution.

  2. The benchmark provides limited support for general medical visual grounding. The released descriptions sometimes contain annotation-derived attributes, including location, shape, and component count. This is not inherently inappropriate, but performance in this setting does not directly establish generalization to independently authored clinical requests. The paper also provides limited evidence that the model can select different targets in the same image when the text changes, or reject mismatched requests. Consequently, the evidence more strongly supports cross-domain segmentation under the evaluated descriptions than text-dependent target selection or complex referring-expression resolution.

  3. Although LaSE requires only one MedSAM2 decoder call, Table 6 reports approximately 4.35B total parameters and 0.773 seconds per image, compared with 0.049 seconds for MedCLIPSeg. Thus, reduced decoder overhead does not establish full-system efficiency. The efficiency claim should be limited to the specific decoding design and supported by matched timing comparisons.

Questions for authors
  1. A hidden-state-only weighting control using the same soft multi-layer fusion mechanism, together with a uniform-fusion baseline, is needed to distinguish the contribution of regional evidence from that of the fusion strategy.
  2. Performance after removing annotation-derived spatial and shape attributes, together with generic-prompt or mismatched-prompt controls, would clarify how strongly the results depend on the target descriptions.
  3. The authors should clarify whether the test sets contain different target requests for the same image and report separate results for such cases or empty-target cases to substantiate target-selection capabilities.
  4. The authors should clarify the scope of the efficiency claim and provide a matched runtime breakdown for the multimodal backbone, layer-wise fusion, and mask decoding, including comparison with the layer-first variant.
Reviewer_7ThiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work targets medical referring visual grounding, where existing segmentation‑token models select transformer layers relying purely on hidden states without inspecting actual spatial localization outputs, leading to noisy maps and background artifacts. The authors propose LaSE, adopting a map‑first paradigm. Its LTM module generates candidate spatial maps from every transformer layer. MEGF adaptively assigns case‑wise layer weights by validating each map against frozen MedSAM2 visual features. A lightweight LMR refinement module upgrades mask quality with only one MedSAM2 decoder invocation. Evaluated on the Source‑4/OOD‑10 medical benchmark, LaSE obtains 90.20% source‑domain DSC and 84.14% out‑of‑distribution DSC, outperforming MedCLIPSeg. Ablations and pre‑decoder soft‑Dice metrics verify that map‑first fusion yields better intermediate spatial cues, and qualitative cases show improved lesion coverage across multiple imaging modalities.

Critical strengths
  1. Instead of picking layers from hidden representations, LaSE generates spatial maps for all layers first and weights them using real medical visual evidence, addressing background contamination in segmentation‑token pipelines.
  2. LTM exposes layer‑wise spatial responses; MEGF uses MedSAM2 regional features for adaptive weighting; LMR brings high‑resolution refinement with a single decoder call.
  3. Significant OOD improvement(+5.12 DSC over SOTA MedCLIPSeg) across ultrasound, dermoscopy, brain‑MRI, endoscopy. Pre‑decoder Soft‑Dice metric quantitatively validates intermediate map quality.
  4. Fair benchmark following MedCLIPSeg protocol, thorough ablations, quantitative map diagnostics, rich qualitative visualizations, and public code for reproducibility.
Critical weaknesses

1.All test prompts are benchmark‑provided descriptions derived from dataset annotations; performance on fully open‑ended real‑world clinical text prompts is unvalidated. 2. The full pipeline has large total parameters and heavy FLOPs dominated by the large MedGemma backbone, limiting practical deployment. 3.Missing comparisons against other multi‑layer aggregation strategies (e.g. simple average / attention fusion without MEGF medical‑feature validation). 4. Occasional disconnected off‑target predictions exist in qualitative examples, yet there is no quantitative breakdown of error modes for OOD samples. 5. Experiments are confined to this fixed benchmark; performance on other public medical referring‑grounding datasets remains unknown.

Questions for authors
  1. How would LaSE behave given fully open‑ended free‑text clinical prompts, instead of annotation‑derived benchmark prompts? Could you add small‑scale tests with manually written clinical queries?
  2. Your model incurs substantial compute cost dominated by the large MedGemma backbone. Are there feasible ways to reduce inference overhead for real‑world deployment, e.g. backbone distillation or layer pruning?
  3. Could you add baseline ablations using simpler multi‑layer fusion schemes (uniform averaging, vanilla attention) without the MEGF medical‑evidence branch, to better isolate MEGF’s unique contribution?
  4. Please quantitatively analyze failure cases on OOD data. What are the main error sources: ambiguous lesion boundaries, misleading text prompts, or domain gap of visual features?
  5. Why do you only evaluate on the MedCLIPSeg benchmark? Could you report results on one additional public medical visual‑grounding dataset to demonstrate broader generalizability?
  6. The learned layer weight β varies between source and OOD data. What properties of input images or text prompts drive these weight shifts? Could you give further interpretive analysis?
Reviewer_dtBbOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes LaSE, a medical visual grounding framework built on a "map-first" principle: instead of selecting a Transformer layer from hidden states before constructing its spatial map, LaSE first constructs a candidate 16×16 spatial map at every one of the L=34 MedGemma layers, then pools response-weighted regional features from a frozen MedSAM2 encoder for each candidate (MEGF) and predicts case-dependent sparse layer weights via entmax-1.5 over a learned temperature. The fused map and fused [SEG] state prompt a single MedSAM2 decoder call, followed by a 29,799-parameter convolutional refinement head (LMR). On the MedCLIPSeg benchmark (4 source / 10 OOD datasets), LaSE reports 90.20% Source-4 and 84.14% OOD-10 DSC vs. 89.11/79.02 for MedCLIPSeg (+1.09/+5.12, best on 12 of 14 tests), improving the pre-decoder map Soft Dice from 54.55% to 59.92% against a layer-first control; ablations attribute 1.60 (MEGF) and 1.33 (LMR) OOD-10 points.

Critical strengths
  1. Clean target-domain isolation and honest reporting. Target images/masks are never used for fitting, selection, or threshold tuning; negative results are reported (loses on BUSI/ISIC sources, a false [REJ], off-target residue in BUID), and even the efficiency table reporting the method's own worst-in-class cost is included (Table 6).
  2. A diagnostic that decouples spatial-prompt quality from decoder capacity. The pre-decoder map-first vs. layer-first comparison and the layer-weight analysis directly probe the central claim instead of relying only on final-mask scores; all headline numbers are internally consistent.
Critical weaknesses
  1. Efficiency claims contradict the paper's own measurements. §5 calls LaSE "efficient … with substantially reduced computational overhead," yet Table 6 shows the highest cost of all methods (0.7729 s/case ≈ 15.7× MedCLIPSeg; 15,293 G FLOPs ≈ 83×; 4,345M params ≈ 20×).
  2. The headline margin is confounded and possibly protocol-inflated, and the paper never decomposes it. The layer-first control (w/o MEGF, 82.54 OOD-10) already exceeds MedCLIPSeg (79.02) by 3.52 points, so most of the +5.12 gain predates the titular mechanism, whose 1.60-point contribution is measured by an ablation that changes three variables at once (decision basis, soft-vs-hard fusion, supervised-vs-REINFORCE optimization).
Questions for authors
  1. How do you reconcile "efficient … substantially reduced computational overhead" with Table 6 ?
  2. Please report per-dataset empty-GT counts, the non-empty-only macro DSC for LaSE and a locally rerun MedCLIPSeg, and how the published baselines handle empty cases under the same double-empty convention. What share of the +5.12 OOD-10 margin survives on non-empty cases only?
  3. Please add the missing ablation cells: uniform-weight fusion (β=1/34); evidence-free soft fusion (β predicted from [SEG] states only, no MedSAM2 regional evidence); evidence-scored hard layer selection; and supervised vs. REINFORCE training for the layer-first variant—with per-dataset OOD breakdowns.
Reviewer_jcvWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes LaSE for medical visual grounding. It constructs spatial similarity maps at each MedGemma layer, uses regional MedSAM2 features to predict fusion weights, and feeds the fused spatial and semantic prompts into a mask decoder followed by lightweight refinement. Experiments on four source and ten target datasets report improvements over published segmentation baselines and a layer-first control.

Critical strengths
  1. Using candidate spatial maps to inform layer fusion is a reasonable extension of segmentation-token grounding. Sharing the MedSAM2 representation across candidates provides a practical way to incorporate regional image evidence.

  2. The evaluation covers multiple imaging modalities and includes both final-mask performance and pre-decoder map diagnostics, providing useful evidence beyond aggregate segmentation scores.

Critical weaknesses
  1. Appendix A.4 applies BCE-with-logits to an already bounded map and computes Dice and boundary losses after sigmoid, restricting their inputs to approximately [0.5, 0.731].

  2. The released entmax implementation produces gradients inconsistent with numerical differentiation, raising concerns about how the core fusion module is trained.

  3. The generation–replay pipeline discards padding masks, potentially affecting batched validation and checkpoint selection.

  4. The spatial-map loss treats bounded responses as logits, leaving its optimization rationale unclear.

  5. The released MEGF-disabled path does not match the layer-first ablation described in the paper, making the central comparison difficult to verify.

Questions for authors
  1. The custom entmax implementation in segmentation_router/modules.py produces gradients that do not match finite-difference checks. Was this implementation used for the reported experiments? If so, does correcting the gradient calculation change the learned layer weights or segmentation results?

  2. Why does the replay step discard the padding mask? Does correcting this change the validation scores or the selected checkpoint?

  3. The raw-dot path in adapter.py produces maps in [0,1] and sets the additional map temperature to one, while losses.py (lines 708–713) applies BCE-with-logits and sigmoid-based Dice and focal losses. Consequently, responses used by these losses lie in approximately [0.5, 0.731], even when the spatial map is zero on background pixels. Is this an intentional constrained-logit objective? Please explain its rationale and whether a consistent probability/logit formulation changes the conclusions.

  4. Section 4.3 describes “w/o MEGF” as a REINFORCE-trained single-layer policy. However, disabling the composer in adapter.py (lines 505–507) produces uniform averaging over all layers, and runtime.py replaces the original selection module with nn.Identity(). Which implementation and configuration produced the layer-first results in Tables 2 and 3? Please identify the corresponding training and evaluation paths so that this comparison can be verified.

Reviewer_mt6gOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes LaSE, a medical visual grounding framework that constructs spatial matching maps from segmentation-token and image-token representations at multiple MedGemma layers. Medical Evidence-Guided Fusion (MEGF) uses map-weighted regional features from a frozen MedSAM2 encoder to predict layer-fusion weights. The resulting spatial and semantic prompts are decoded through a single MedSAM2 decoder call, followed by Lightweight Mask Refinement (LMR).

Critical strengths
  1. A concrete and well-motivated research question. The paper focuses on converting intermediate multimodal representations into effective spatial prompts. Using regional image evidence to guide layer fusion is a plausible design.

  2. Consistent improvements across target datasets. LaSE outperforms the reported MedCLIPSeg results on all ten target datasets, rather than obtaining its aggregate improvement from a single target.

  3. Evaluation of pre-decoder spatial evidence. OOD-10 map SoftDice improves from 54.55% to 59.92%, suggesting that the gains are not entirely attributable to final-mask refinement.

  4. Detailed disclosure of the experimental protocol. The appendix documents prompt provenance, source-only checkpoint selection, rejection handling, trainable components, computational measurements, and the interpretation of bootstrap intervals.

Critical weaknesses
  1. The central ablation does not isolate the contribution of regional spatial evidence. The “w/o MEGF” variant replaces region-conditioned multi-layer fusion with hidden-state-based single-layer selection trained using REINFORCE. This comparison simultaneously changes the evidence available to the selector, single-layer versus multi-layer composition, and the optimization method.

  2. System-level improvements are not fully separated from the proposed mechanism. External comparisons use published results, while LaSE also changes the foundation models, resolution, decoder, and auxiliary supervision. Table 6 reports approximately 4.345B total and 678.7M trainable parameters, substantially more than MedCLIPSeg. This limits the interpretation of the 5.12-point improvement to the complete system.

  3. Annotation-derived descriptions limit the scope of the grounding claims. Appendix A.2 discloses that some input descriptions contain location, count, and shape attributes derived from labels or masks. Using the same benchmark is reasonable, but these results do not directly establish generalization to independently authored clinical requests. Similarly, perfect [SEG]/[REJ] decisions may partly reflect target-presence information in the text rather than visual evidence.

  4. The overall efficiency claim lacks a matched comparison. LMR’s small parameter count and the single mask-decoder call are useful local properties. However, the complete system still requires generation and replay through a large multimodal backbone. The reported timing measurements are not hardware-matched, and no direct comparison establishes an efficiency advantage over relevant multi-scale or multiple-decoder alternatives using the same backbone.

Questions for authors
  1. What happens when the candidate maps, multi-layer fusion, model capacity, and optimization are preserved, but regional evidence is removed or replaced by global pooling? Can fixed fusion weights achieve comparable performance?

  2. Within the same MedGemma–MedSAM2 system, are the gains over reasonable fixed-layer and multi-layer fusion controls consistent across training runs? Please report paired differences and training variability for the key comparison.

  3. Does LaSE retain its advantage when annotation-derived location, count, and shape cues are removed? How well does a text-only baseline perform on the [SEG]/[REJ] decision?

  4. Can the efficiency claim be supported by a comparison using the same backbone and hardware, or narrowed to LMR’s incremental overhead and the use of a single mask-decoder call? Providing experimental results regarding FLOPs and inference time would be more convincing.

  5. Why are there so many non-trivial hyperparameters involved in "Map normalization and training losses", and Equation 13? Has this been experimentally verified?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.