acceptodds
Withdrawn from ICLR 2027

RFMedSAM 2: Automatic Prompt Refinement for Medical Image Segmentation with SAM 2

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The Segment Anything Model 2 (SAM 2) is a prompt-driven foundation model that generalizes SAM to both image and video segmentation, achieving strong zero-shot segmentation performance. However, despite its potential for medical image analysis, SAM 2 inherits key limitations from its predecessor, including binary mask outputs, inability to predict semantic labels, and reliance on precise prompts for target localization. Moreover, its direct application to medical image segmentation remains suboptimal. To investigate its upper performance bound, we fine-tune SAM 2 with targeted architectural modifications and oracle (ground-truth) prompts, reaching a Dice scores of 92.3% on BTCV, surpassing the state-of-the-art nnUNet by 9.8%; we stress that this is a diagnostic upper bound, not a comparable operating point, since it consumes ground-truth boxes and class labels. To eliminate the reliance on manual prompts, we develop an automatic prompt generator based on U-Net, which predicts coarse masks and bounding boxes that are subsequently refined through SAM 2’s progressive dual-stage inference. Furthermore, we propose QRefine-Former to transfer domain-specific medical knowledge from the U-Net prompt generator to the SAM 2 foundation model, further improving segmentation performance. Extensive experiments on the AMOS22, ACDC, and BTCV datasets demonstrate the superiority of our method, achieving Dice scores of 91.0% and 88.6% on AMOS22 Task 1 and Task 2, 93.6% Dice on ACDC, and 86.8% Dice on BTCV.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
SAM 2, Medical Image Segmentation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_zbpNOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper first evaluates the segmentation performance of SAM/SAM 2 on medical images under oracle settings and investigates their preferences for different prompt modes. It then proposes RFMedSAM 2, which first uses a UNet to generate masks for prompt generation and subsequently leverages SAM 2 to refine the segmentation using inter-frame relationships. To bridge UNet and SAM 2, the authors further introduce an adapter designs termed QRefine-Former. However, this paper reads more like an evaluation report and engineering adaptation of SAM/SAM 2 to medical domain. The technical novelty is limited, the effectiveness of the proposed method is not sufficiently validated, and the emphasized SAM-based refinement brings only marginal improvements over the UNet predictions. Therefore, I do not think this paper is suitable for publication at ICLR.

Critical strengths
  1. The paper provides a detailed evaluation of SAM/SAM 2 under different prompt modes for medical image segmentation, which may provide useful information for future work.

  2. The engineering organization and integration of the proposed framework are reasonable.

Critical weaknesses
  1. The technical innovation is quite limited. Adapting SAM/SAM 2 to volumetric medical image segmentation has already been widely studied, including approaches with automatic prompt generation [1,2,3]. It is therefore unclear what substantial advantages the proposed method offers over these existing approaches.

  2. The contribution of the proposed QRefine-Former also appears limited. For example, the DSC improves only from 0.867 to 0.868 on BTCV, from 0.907 to 0.910 on AMOS-CT, and from 0.884 to 0.886 on AMOS-CTMRI. Improvements of this magnitude could potentially result from experimental randomness. It is unclear whether these improvements are robust and whether QRefine-Former is effective.

  3. I am also concerned that, according to Table 6, SAM 2 provides only marginal refinement over the UNet segmentation, with improvements of 0.003 DSC on ACDC, 0.005 on BTCV, and 0.002 on AMOS-CTMRI. This suggests that SAM 2 contributes very little to the final segmentation performance on these datasets, while introducing substantial additional model parameters and computation compared with U-Net alone. So why is SAM 2 even needed in this framework?

  4. The authors claim that their method provides efficient adaptation. However, the efficiency analysis is mainly limited to parameter counts. Computational costs such as FLOPs, GPU memory consumption, and inference speed are not reported, although these metrics are important for practical applications. I suggest that the authors provide these results and compare them with other baselines.

[1] Xing Y, Wu J, Bu Y, et al. SAM2-SGP: Enhancing SAM2 for medical image segmentation via Support-Set Guided Prompting[J]. arXiv preprint arXiv:2506.19658, 2025. [2] Li C, Ibn Sultan R, Khanduri P, et al. Autoprosam: Automated prompting sam for 3d multi-organ segmentation[C]//Proceedings of the Winter Conference on Applications of Computer Vision. 2025: 3570-3580. [3] Xie B, Tang H, Duan B, et al. Masksam: Towards auto-prompt sam with mask classification for volumetric medical image segmentation[J]. arXiv preprint arXiv:2403.14103, 2024.

Questions for authors

Please see weakness for the questions.

Reviewer_ZFqMOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary
  • The paper proposes RFMedSAM 2, a framework that adapts SAM 2 for fully automatic medical image segmentation.
  • It uses a U-Net to automatically generate mask and bounding-box prompts, followed by dual-stage refinement with SAM 2. A QRefine-Former module transfers medical-specific features from U-Net to SAM 2.
  • Extensive experiments on AMOS22, ACDC, and BTCV demonstrate competitive segmentation performance compared with existing medical segmentation methods.
Critical strengths
  • The method integrates automatic prompt generation with SAM 2, reducing the need for manual prompts in medical image segmentation.
  • The authors conduct comprehensive evaluations across multiple datasets and imaging modalities, with ablation studies to validate the effectiveness of different components.
Critical weaknesses
  • Limited Novelty and Missing Comparisons with Recent Methods: Several recent studies have explored automatic prompting and medical-domain adaptation of SAM/SAM 2, including AutoProSAM (WACV 2025), OFL-SAM2 (AAAI 2026), and GU-SAM2 (MICCAI 2026). In particular, GU-SAM2 also transfers U-Net-based medical representations into SAM 2, raising questions about the novelty of QRefine-Former. The paper lacks sufficient discussion and comparisons with these closely related approaches.
  • Insufficient Justification for the Refinement Pipeline: Since the U-Net already produces segmentation masks, the necessity of further refinement through SAM 2 remains unclear. On BTCV, the final prediction improves only marginally over the U-Net output (86.3% to 86.8% Dice), despite introducing substantial additional model complexity and computational overhead. The limited improvement raises concerns about whether the refinement pipeline is justified.
  • Unclear Necessity of Semantic Label Inference: The paper identifies SAM 2's inability to predict semantic labels as a key limitation. However, introducing task-specific semantic prediction may compromise the open-world generalization capability of foundation models.
Questions for authors
  • What are the key methodological advantages of RFMedSAM 2 over these recent methods, particularly in automatic prompting and medical feature fusion?
  • Can the authors provide comparisons with relevant recent baselines under consistent evaluation settings, especially on shared benchmarks such as BTCV?
  • Does SAM 2 consistently correct errors in the initial U-Net predictions, particularly when the generated masks are inaccurate?
Reviewer_D4bUOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes RFMedSAM 2 for automatic volumetric medical image segmentation, combining a 3D U-Net prompt generator, adapted SAM 2, two-stage mask refinement, and QRefine-Former for feature transfer.

Critical strengths

The framework removes the need for manual prompts and reports competitive segmentation results across abdominal CT and cardiac MRI benchmarks. The investigation of different prompt-generation strategies is useful.

Critical weaknesses
  1. The accuracy–efficiency trade-off is not established. The pipeline combines a 3D U-Net, SAM 2 feature extraction, two mask-prediction stages, memory processing, and feature fusion, but reports no end-to-end latency, throughput, or peak GPU memory. The complete pipeline improves over the jointly trained U-Net by only 0.2–0.9 Dice percentage points in Table 6. Without computational measurements, it remains unclear whether these gains justify the additional inference complexity.

  2. The independent benefit of cross-slice memory is insufficiently isolated. Although Figure 4 and Table 6 provide useful ablations, the BTCV improvement from Step 1 to Step 2 is only 0.1 percentage points. This transition also introduces updated prompts and an additional decoding pass, so the improvement cannot be attributed solely to memory. A matched per-slice refinement baseline would help establish the specific benefit of cross-slice memory and justify its additional complexity.

Questions for authors
  1. Could the authors report end-to-end inference latency, throughput, and peak GPU memory for the complete system and the jointly trained U-Net alone, using identical hardware and input settings?
  2. Table 6 reports a BTCV improvement from 86.7% at Step 1 to 86.8% at Step 2. How much of this gain is specifically attributable to cross-slice memory? Could the authors compare against a matched two-pass, per-slice refinement baseline using the same prompt updates but without cross-slice memory?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.