acceptodds
Under review as a conference paper at ICLR 2027

SegBanana: Steering Unified Multimodal Models into Medical Segmenters

Abstract

Medical image segmentation remains challenging in practical deployment, as models often struggle to generalize beyond the distributions covered by their training data and high-quality pixel-level annotations are typically unavailable for adaptation. Inspired by the cross-task transferability of large language models, we investigate whether unified multimodal models (UMMs) can transfer their pretrained visual understanding, reasoning, and generation capabilities to medical image segmentation without task-specific post-training. By recasting segmentation as structured visual generation, we find that frontier UMMs (e.g., Nano Banana) already exhibit basic segmentation capabilities across diverse clinical scenarios, but still struggle with challenging tasks requiring specialized anatomical or domain-specific knowledge. We further show that these limitations can be effectively mitigated by incorporating visual anatomical knowledge from in-context exemplars, expanding candidate solutions through repeated sampling, and refining suboptimal predictions via targeted editing. Motivated by these observations, we propose SegBanana, to our knowledge, the first agentic visual generation framework for training-free medical image segmentation. SegBanana builds on a frozen UMM as the core generative model, augmented with Anatomy-Aware Knowledge Retrieval and Comparative Quality Critique to unlock its potential segmentation capability. A State-Aware Multimodal Controller maintains structured state and iteratively orchestrates these tools, repeatedly refining intermediate predictions toward higher-quality masks. Across eight medical segmentation datasets, SegBanana achieves an average mDice of , outperforming representative generalist (SAM3 and SegGPT) and medical-specific (BiomedParse and MedSAM3) baselines by at least points, while remaining robust to out-of-domain visual supports. Ablations further confirm the effectiveness of agentic inference and the contributions of retrieval, critique, and structured state.

Primary area:
generative models
Keywords:
Generative Models, Medical Image Segmentation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_1f6KOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether pretrained multimodal models can perform medical image segmentation without task-specific training. It proposes SegBanana, which combines a frozen image generation model with retrieval of annotated examples, candidate comparison, and a controller that tracks previous attempts. Based on feedback, the controller decides whether to generate a new mask, refine an existing one, or stop. Experiments on eight datasets report an average mDice of 77.45%, compared with 57.69% for direct generation, exceeding the evaluated baselines on average.

Critical strengths
  1. Clear and informative visualizations. The framework overview and step-by-step case study make the inference process easy to follow. The qualitative comparisons also illustrate how the method handles different anatomical structures and segmentation errors.

  2. Well-motivated method design. The preliminary experiments identify limitations of direct generation and examine how retrieved examples, diverse sampling, and targeted editing can improve results. These observations provide a clear rationale for the proposed components.

Critical weaknesses
  1. Unclear source of improvement. Without a fixed workflow matched for sampling budget, the gains from adaptive control cannot be separated from those of additional computation.

  2. Insufficient baseline details. Near-zero results for some baselines warrant closer validation of prompts, preprocessing, and output handling.

  3. Inconclusive ablations. The critic ablation retains evaluation within the controller, and the state ablation retains full history, limiting conclusions about their necessity.

  4. Unquantified critic reliability. Incorrect candidate selection and premature stopping are not measured, despite the gap between selected and oracle results.

  5. Incomplete reproducibility and cost reporting. Sample counts, support-pool construction, repeated-run variability, and total inference cost are insufficiently documented.

Questions for authors

The framework currently relies on Nano Banana 2 as its UMM backbone. Can it be adapted to other UMMs, such as GPT-image2?

4 comments

We sincerely thank the reviewer for the thoughtful review and for highlighting the clarity of our visualizations and the role of the preliminary experiments in motivating SegBanana's design. We also appreciate the suggestions for isolating the contribution of adaptive control under a matched sampling budget, strengthening the baseline and ablation studies, and quantifying the critic's reliability.

Below, we address the comments and question point by point.

Weakness 1: Source of Improvement Under a Fixed Sampling Budget

Reviewer Question: "Unclear source of improvement. Without a fixed workflow matched for sampling budget, the gains from adaptive control cannot be separated from those of additional computation."

Response: We add a fixed-budget sampling baseline. For each dataset, we rounded up SegBanana's average number of generator calls and used that integer as the baseline's budget for every image. We applied the same oracle criterion to all methods, scoring the best candidate mask for each image against the ground truth. SegBanana reached 81.06, outperforming the fixed-budget baseline (66.86) on all eight datasets by an average of 14.20 Dice points while using fewer generator calls on average (3.93 vs. 4.50 per image). Thus, SegBanana achieved higher Dice with fewer generator calls, highlighting the benefit of adapting refinement to each sample's difficulty.

MethodTNBCRAVIRDrishti-GSISICBUS-UCLMKvasirACDCBraTSMean
Nano Banana 2 Dice67.0434.4443.7374.6869.3085.0636.8950.3857.69
Strict budget555444544.50
Strict-budget Dice70.9952.6065.5683.2973.6486.4042.6059.8066.86
SegBanana avg. generator calls/image4.724.504.763.243.183.124.523.403.93
SegBanana oracle Dice77.9981.8392.7987.5282.5189.7465.9470.1781.06

Weakness 2: Validation of Near-Zero Baseline Scores

Reviewer Question: "Insufficient baseline details. Near-zero results for some baselines warrant closer validation of prompts, preprocessing, and output handling."

Response: We checked the prompts, preprocessing, and conversion of model outputs to evaluation masks for all baselines with near-zero scores and found no implementation errors in these steps. Among the medical-specific methods, low scores were task-specific. MedSAMAgent scored 0.30 mDice on RAVIR and 83.47 on ISIC, while IBISAgent scored 1.00 on TNBC and 87.47 on ISIC. This variation suggests difficulty generalizing across imaging modalities and target structures. Among the general-purpose segmentation models, SAM 3 scored 56.65 mDice on ACDC and 0.00 on TNBC, RAVIR, Kvasir, and BraTS, where it returned no mask for nearly all target-positive images we inspected. We suspect it has difficulty linking specialized anatomical terms to the corresponding image structures. Kong et al. [1] similarly found that text prompts alone often failed to identify nuclei in pathology images.

[1] Kong Q, Sharifi S, Wang Y, et al. Is SAM3 ready for pathology segmentation?[J]. arXiv preprint arXiv:2604.18225, 2026.

Weakness 3: Contribution of the Independent Critic and Explicit State

Reviewer Question: "Inconclusive ablations. The critic ablation retains evaluation within the controller, and the state ablation retains full history, limiting conclusions about their necessity."

Response: Without evaluation from either the critic or the controller, SegBanana has no feedback to guide refinement and can only sample repeatedly. As reported in Weakness 1, fixed-budget repeated sampling achieved 66.86 mean oracle Dice, compared with 81.06 for SegBanana, despite using more generator calls per image (4.50 vs. 3.93). To isolate the contribution of the independent critic, we therefore retained refinement and assigned candidate evaluation to the controller. Under this ablation, mean Dice fell from 77.45 to 72.33. Similarly, the explicit-state ablation retained the full interaction history so the agent could still use prior feedback. We will clarify in the main text what the independent critic adds beyond controller self-evaluation and what the explicit state adds beyond access to the full history.

Weakness 4: Candidate Selection and Premature Stopping

Reviewer Question: "Unquantified critic reliability. Incorrect candidate selection and premature stopping are not measured, despite the gap between selected and oracle results."

Response: Candidate selection: In approximately 62% of cases, the selected mask’s Dice score was within 0.5 points of the oracle score. In only 7% of cases did the gap exceed 5 points. Averaged equally across the eight datasets, selected and oracle mDice were 77.45% and 81.06%, respectively, a selection loss of 3.61 points.

Premature stopping: The controller stopped at the first iteration in 46.0% of cases, at the second in 14.0%, and at the third in 40.1%. To assess whether continuing would improve the result, we forced one additional EDIT after stopping on the premature samples. The additional EDIT step produced only small changes in dataset-level mean mDice on these cases.

MetricTNBCRAVIRDrishti-GSISICBUS-UCLMKvasirACDCBraTS
Selected mDice (%)74.7580.9387.0982.0678.0789.5361.6865.52
Oracle mDice (%)77.9981.8392.7987.5282.5189.7465.9470.17
Selection loss (pp)3.240.905.705.464.440.214.264.65
mDice at stopping (%)73.8981.8963.7388.8583.7689.6276.6977.24
mDice after extra EDIT (%)73.8481.9264.5689.6084.2089.6877.0276.40
Δ mDice (pp)−0.05+0.03+0.83+0.75+0.44+0.06+0.33−0.84

Weakness 5: Reproducibility and Inference Cost

Reviewer Question: "Incomplete reproducibility and cost reporting. Sample counts, support-pool construction, repeated-run variability, and total inference cost are insufficiently documented."

Response: Repeated-run variability. Across three independent runs on the same test cases, the mean macro-averaged mDice was 76.79%, with a sample standard deviation of 0.63 percentage points and a sample variance of 0.40 squared percentage points.

MethodSettingTNBCRAVIRDrishti-GSBUS-UCLMKvasirACDCISICBraTSAvg.
BiomedParse-11.815.6778.1963.8188.9589.0387.5975.1162.52
MedSAM3-68.214.2134.1357.6891.1716.7489.8055.4252.17
SegBananaRun 174.7580.9387.0978.0789.5361.6882.0665.5277.45
SegBananaRun 272.6177.3587.9576.5588.2860.6782.6963.4576.19
SegBananaRun 369.0580.4586.5976.6489.0660.8685.4065.8376.74
SegBananaMean ± SD72.14 ± 2.8879.58 ± 1.9487.21 ± 0.6977.09 ± 0.8588.96 ± 0.6361.07 ± 0.5483.38 ± 1.7764.93 ± 1.2976.79 ± 0.63

Evaluation splits, sample counts, support-pool construction, and inference cost: We used the original splits when test annotations were public. Otherwise, we held out at least 20% of labeled data, splitting by patient when identifiers were available. For 3D MRI, we randomly sampled two slices per patient. In total, we evaluated 1,945 images or slices across the eight datasets. At each retrieval step, we searched all annotated image–mask pairs in the target training set (in-domain) or a training set matched by modality and task (OOD), selecting two pairs based on query similarity and diversity. The supplement lists the support dataset for each target. For cost estimation, we rounded up the average numbers of generator calls and iterations per image to 4 and 2, respectively. Assuming approximately five Gemini calls per iteration for the controller and critic combined, this gives an estimated 10 Gemini calls per image. At USD 0.08 per generator call and USD 0.02 per Gemini call, the estimated cost was USD 0.52 per image.

Question 6: Alternative UMM Backbone

Reviewer Question: "The framework currently relies on Nano Banana 2 as its UMM backbone. Can it be adapted to other UMMs, such as GPT-image2?"

Response: Yes. We evaluated SegBanana with GPT-image-2 as the generator across all eight datasets. Mean Dice increased from 49.13 for the standalone generator to 68.87 with SegBanana, a gain of 19.74 Dice points.

MethodTNBCRAVIRDrishti-GSISICBUS-UCLMKvasirACDCBraTSAvg.
BiomedParse11.815.6778.1987.5963.8188.9589.0375.1162.52
MedSAM368.214.2134.1389.8057.6891.1716.7455.4252.17
GPT-image-2 Dice37.0432.7345.8363.6044.5980.5038.0850.6449.13
SegBanana (GPT-image-2) Dice53.6773.4976.4077.9767.3287.5962.7651.7868.87

Thank the authors for the response. Good paper! I keep my score unchanged.

Thank you for your thoughtful assessment and detailed feedback. Your suggestions prompted us to add a fixed-budget baseline and examine how well SegBanana selects masks and decides when to stop. We will include these results in the revised manuscript and describe the evaluation protocol more clearly.

Again, we sincerely appreciate the time you devoted to reviewing our work.

Reviewer_gvFTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Uses a frozen Nano Banana 2 as a medical segmenter by prompting it for a colour-coded label map and decoding by colour. Around it sits an agent loop: DINOv3+MMR retrieval from an annotated support pool, Generate/Edit actions, a frozen VLM critic doing pairwise comparison, and Gemini-3.5 as a controller with a compact state, up to three iterations. Averages 77.45 mDice over eight 2D medical datasets, above BiomedParse (62.5) and SegGPT (52.3). Ablations attribute ~5 points each to retrieval, critic and state; raw Nano Banana 2 gets 57.7. Also reports controller swaps, out-of-domain supports, inference cost, and critic-vs-oracle gaps.

Critical strengths

The result is good: no parameter updates, 77 mDice average across eight modalities, 25 points above the best few-shot generalist, and each component removal costs ~5 points (Table 3). Even discounted, the gap is not noise. Honest diagnosis: Fig. 5 shows critic-selected masks stay below the oracle at every iteration and names candidate selection, not the generator, as the bottleneck. The appendix gives full prompts, per-sample call counts and stopping statistics.

Critical weaknesses

1.Table 2 is not a like-for-like comparison, and the paper does not say so. SegBanana uses annotated supports from the same dataset (retrieval fires on 83% of samples, 100% on four datasets, Table 8), yet the headline gap is against zero-shot BiomedParse and MedSAM3. 2.Fig. 4 gives no numbers

Questions for authors
  1. Please turn Fig. 4 into a table, add ISIC and BraTS, include BiomedParse / MedSAM3.
  2. Can you address contamination, e.g. by testing on a dataset released after the generator’s training cutoff? Please also give model versions and dates, and variance over at least three runs.
6 comments

We sincerely thank the reviewer for commending our strong segmentation results, effective workflow components, and clear presentation. We also appreciate the reviewer's identification of shortcomings in the clarity of comparison settings, quantitative reporting, and validation of potential data contamination and run-to-run variability. We address these concerns below.

Weakness 1: Comparison Setting

Reviewer Question: "Table 2 is not a like-for-like comparison,... SegBanana uses annotated supports from the same dataset, yet the headline gap is against zero-shot BiomedParse and MedSAM3. "

Response: Table 2 was designed to compare how different segmentation paradigms generalize in practice, following each method's intended usage conditions. We therefore group medical specialists and general-purpose models separately and label their zero-shot/few-shot settings. We agree that the reported 14.93-point gap reflects differences in both method and inference-time supervision. SegBanana can retrieve annotated examples from the target dataset. BiomedParse and MedSAM3 use no annotated examples at inference, although their reported training data cover three and two of our eight benchmarks, respectively.

To match access to annotated support at inference, we disabled retrieval and repeated sampling three times. SegBanana achieved 68.40% mean Dice, compared with 62.52% for BiomedParse and 52.17% for MedSAM3. However, disabling retrieval increased run-to-run variance, underscoring the importance of retrieval for stable segmentation performance.

MethodSettingTNBCRAVIRDrishti-GSBUS-UCLMKvasirACDCISICBraTSAvg.
BiomedParseZero-shot11.815.6778.1963.8188.9589.0387.5975.1162.52
MedSAM3Zero-shot68.214.2134.1357.6891.1716.7489.8055.4252.17
SegBananaw/o Retrieval68.54 ± 6.9975.20 ± 5.8374.85 ± 3.4168.38 ± 8.8583.82 ± 4.2443.69 ± 8.9878.95 ± 3.1353.80 ± 9.0268.40 ± 7.37
SegBanana-72.14 ± 2.8879.58 ± 1.9487.21 ± 0.6977.09 ± 0.8588.96 ± 0.6361.07 ± 0.5483.38 ± 1.7764.93 ± 1.2976.79 ± 0.63

Weakness 2 & Question 1: Expanded OOD Evaluation

Reviewer Question: "Fig. 4 gives no numbers" & "Please turn Fig. 4 into a table, add ISIC and BraTS, include BiomedParse / MedSAM3."

Response: Fig. 4 examined how segmentation performance changes when annotated in-domain support is unavailable and only external support can be retrieved. The original comparison therefore focused on methods that use support examples. BiomedParse and MedSAM3 use no support examples at inference. As requested, we will include both as zero-shot references in the table replacing Fig. 4.

We extended the evaluation to ISIC and BraTS using annotated support examples from PH² [1] and UTSW-Glioma [2], respectively. With external support, SegBanana’s mDice fell by only 0.67 points on ISIC and 2.07 points on BraTS. Across eight datasets, SegBanana’s mean mDice declined by 7.0%, compared with 18.9% for SegGPT, the second-best method using support examples. SegBanana with external support also outperformed its no-retrieval variant by 3.61 mDice points (72.01 vs. 68.40), showing that retrieved examples remain useful even when in-domain annotations are unavailable.

MethodSupportTNBCRAVIRDrishti-GSBUS-UCLMKvasirACDCISICBraTSAvg.
BiomedParse—11.815.6778.1963.8188.9589.0387.5975.1162.52
MedSAM3—68.214.2134.1357.6891.1716.7489.8055.4252.17
UniverSegID24.4828.4540.6813.3819.9121.4839.6415.4825.44
UniverSegOOD29.3928.570.928.0217.7910.2147.7312.3119.37
PainterID22.946.551.728.0527.601.3627.6515.5513.93
PainterOOD21.904.071.806.2127.401.3729.5015.5613.48
SegGPTID53.3364.1976.8232.0865.5745.2051.8129.0852.26
SegGPTOOD17.0061.8862.4525.4265.0116.6355.6335.2142.40
MatcherID25.1721.2435.6138.3853.6720.4855.0927.9434.70
MatcherOOD24.7320.9035.1630.9658.729.4162.1336.4334.81
PerSAMID5.2519.4133.3732.4644.9918.1928.5314.6824.61
PerSAMOOD3.7922.7135.7025.4944.2011.0943.1525.7326.48
GF-SAMID48.9837.9124.4842.0753.0013.1663.6817.2037.56
GF-SAMOOD43.8234.8427.0336.9859.324.0063.3416.0035.67
INSID3ID31.1633.636.2523.7274.494.7863.8721.7232.45
INSID3OOD29.9033.386.2518.9271.875.0069.2819.4731.76
SegBananaID74.7580.9387.0978.0789.5361.6882.0665.5277.45
SegBananaOOD73.5479.2463.6175.0985.2854.4481.3963.4572.01

Question 2: Training-Data Contamination and Run-to-Run Variability

Reviewer Question: "Can you address contamination, e.g. by testing on a dataset released after the generator's training cutoff? Please also give model versions and dates, and variance over at least three runs."

Response: Our image generator is Nano Banana 2 (Gemini 3.1 Flash Image), first released in preview on February 26, 2026, with a reported knowledge cutoff of January 2025. Our controller is Gemini 3.5 Flash (gemini-3.5-flash), released on May 19, 2026.

To assess potential training-data overlap, we evaluated SegBanana on UTSW-Glioma [2] and RCC-AID [3]. TCIA released UTSW-Glioma on March 18, 2026, while RCC-AID was released on April 22, 2026. Both releases followed the generator’s initial preview. SegBanana achieved 74.16 mDice on UTSW-Glioma and 79.13 on RCC-AID, outperforming BiomedParse by 1.75 and 32.83 points, respectively.

MethodUTSW-Glioma mIoUUTSW-Glioma mDiceRCC-AID mIoURCC-AID mDice
BiomedParse60.2972.4138.3946.30
MedSAM311.0517.6020.8824.79
SegBanana62.4574.1669.0379.13

We also evaluated SegBanana on the same test cases in three independent runs. Across the eight benchmarks, the macro-averaged mDice was 76.79%, with a sample variance of 0.4 squared percentage points.

MethodResultTNBCRAVIRDrishti-GSBUS-UCLMKvasirACDCISICBraTSAvg.
BiomedParse-11.815.6778.1963.8188.9589.0387.5975.1162.52
MedSAM3-68.214.2134.1357.6891.1716.7489.8055.4252.17
SegBananaRun 174.7580.9387.0978.0789.5361.6882.0665.5277.45
SegBananaRun 272.6177.3587.9576.5588.2860.6782.6963.4576.19
SegBananaRun 369.0580.4586.5976.6489.0660.8685.4065.8376.74
SegBananaMean ± SD72.14 ± 2.8879.58 ± 1.9487.21 ± 0.6977.09 ± 0.8588.96 ± 0.6361.07 ± 0.5483.38 ± 1.7764.93 ± 1.2976.79 ± 0.63

[1] Mendonça T, Ferreira P M, Marques J S, et al. PH 2-A dermoscopic image database for research and benchmarking[C]//2013 35th annual international conference of the IEEE engineering in medicine and biology society (EMBC). IEEE, 2013: 5437-5440.

[2] Reddy D. The University of Texas Southwestern Glioma MRI dataset with molecular marker characterization and segmentations (UTSW-Glioma)(Version 1)[Data set][J]. The Cancer Imaging Archive, 2026.

[3] de Boer S, Häntze H, Ziegelmayer S, et al. RCC-AID: Renal Cell Carcinoma AI Dataset for Medical Imaging Research[J]. medRxiv, 2026: 2026.04. 22.26351451.

Reviewer_xpa9OpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper show an agentic workflow to make frozen Nano Banana 2 model work on medical image segmentation. The workflow involves retrieving example image/mask pairs, and an iteration that are aiming to improve the segmentation results. The design of the workflow mimics how human is thinking.

The result show a big improvement comparing with direction segmentation and other models .

Critical strengths
  1. Clear writing and figures
  2. The result is promising
  3. The design of the workflow mimics how human is thinking.
  4. The experiment is thoughout
Critical weaknesses

The workflow is not open-sourced

Questions for authors

N/A

3 comments

We sincerely thank the reviewer for the positive assessment of our workflow design, segmentation results, presentation, experimental evaluation, and the technical soundness of our approach.

Regarding code availability, we will publicly release our implementation upon acceptance. We are also working to package the workflow as a reusable agent skill to facilitate broader adoption.

Nice work. I keep my score unchanged.

Thank you very much for your generous assessment and encouragement. We will continue to refine SegBanana to make its workflow easier to use.

Comments on the paper

1 comment

Dear Area Chairs and Reviewers,

We thank all three reviewers for their careful evaluations and constructive feedback on SegBanana. Overall, the reviewers highlight the “Well-motivated method design” and “Clear and informative visualizations” (1f6K), and describe our presentation as “Clear and easy to follow” (xpa9, gvFT, 1f6K) . They also recognize the segmentation results, stating that “The result is promising” (xpa9) and “The result is good” (gvFT). Reviewer xpa9 further commends the human-inspired workflow design and reports “no substantive concerns about the paper’s soundness.”

During the author response period, we conducted additional experiments, clarified the evaluation protocols, and revised our claims and statistical reporting. These additions strengthen our work by better isolating the benefits of adaptive control, extending validation across datasets and generators, and improving reproducibility and transparency. Our main updates are summarized below:

  • We have clarified the three support settings and expanded external-support comparisons, reporting a 5.88-point macro-averaged mDice gain over BiomedParse without retrieval (To gvFT).

  • We have added a fixed-budget sampling baseline, showing that SegBanana achieves 14.20 points higher oracle Dice with fewer generator calls (To 1f6K).

  • We have clarified the critic/state ablations and analyzed selection and stopping, strengthening evidence for component contributions and identifying remaining bottlenecks (To 1f6K).

  • We have evaluated SegBanana with GPT-image-2 across eight datasets, improving mean Dice from 49.13 to 68.87 over the standalone generator (To 1f6K).

  • We have added three-run evaluations on UTSW-Glioma, RCC-AID, and MedPelvis3D, strengthening robustness evidence across additional datasets and anatomical targets (To gvFT and 1f6K).

  • We have documented evaluation protocols, support pools, model versions, and estimated inference costs, improving reproducibility and transparency (To gvFT and 1f6K).

  • We have committed to releasing the implementation upon acceptance and are developing a reusable agent skill to facilitate reproduction and broader adoption (To xpa9).

Detailed answers and supporting experimental results are provided in our individual responses. Further discussions arewelcomed to facilitate the reviewing process toward a comprehensive evaluation of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.