acceptodds
Under review as a conference paper at ICLR 2027

Segmentation as Editing: Unified Medical Optical Image Segmentation with Differentiable Edit-Space Geometry Adaptation

Abstract

Medical optical imaging is widely used in clinical diagnosis and pathological assessment. Precise segmentation of clinically meaningful anatomical structures and pathological regions enables quantitative assessment for clinical decision-making and research. However, this field still faces at least two challenges: the lack of a unified segmentation dataset dedicated to diverse optical modalities and the need for a unified segmentation model that achieves competitive performance across multiple optical imaging modalities. To this end, we construct OptiMedEdit-37K, a segmentation-centered medical optical image editing dataset covering 7 medical optical imaging modalities and 18 anatomical and pathological targets. Meanwhile, we develop Segmentation as Editing as a unified paradigm for medical optical image segmentation. Through staged fine-tuning on OptiMedEdit-37K under this paradigm, we first establish a Unified Medical Optical image segmentation model (UniMedOpti). To retain its shared capabilities while better modeling target-specific geometric structures, we then augment UniMedOpti with Geometry-aware Residual Adaptation through Differentiable Edit-space Alignment (GRADE). It freezes the shared backbone and trains low-rank residuals using geometric losses on soft masks differentiably recovered from generated edits. UniMedOpti achieves competitive segmentation performance across multiple medical optical imaging modalities and datasets. The experimental results demonstrate that Segmentation as Editing combines cross-modal generality with competitive accuracy.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Medical image segmentation, Multimodal learning, Diffusion models, Image editing, Unified segmentation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_9JfEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Segmentation as Editing, a framework that formulates medical image segmentation as instruction-guided image editing. The authors construct OptiMedEdit-37K across seven optical imaging modalities and fine-tune Qwen-Image-Edit for unified segmentation. They further introduce GRADE, which combines LoRA with geometric supervision to improve vessel segmentation.

Critical strengths

The experiments cover diverse imaging modalities and segmentation tasks and show consistent improvements from geometric supervision

Critical weaknesses
  1. The novelty is somewhat limited, and I don't find the idea particularly interesting.

  2. The paper lacks comparisons with conventional segmentation models such as nnU-Net, and SAM/MedSAM.

  3. The direct-mask baseline still uses a generative model. It remains unclear whether image editing offers advantages over conventional discriminative segmentation.

  4. The method relies on a 20B generative model, but training costs, inference latency, and throughput are not reported or compared with segmentation baselines.

  5. Most evaluations use held-out samples from datasets included in training. Experiments on unseen datasets or modalities would better support the claim of unified segmentation.

Questions for authors

see weakness

4 comments

I have not yet received or seen any response from the authors and would appreciate clarification on the questions I raised.

Thank you for your time and valuable feedback.

W1-W3 Motivation, novelty, and comparison with discriminative segmentation.

Our goal is a single reusable segmentation model for heterogeneous medical optical imaging. As modalities and target structures multiply, maintaining a separate model for each task becomes increasingly burdensome, and annotations differ in definition and format across datasets.

We address data and model adaptation together. Our curated corpus organizes the existing image-mask annotations scattered across datasets into a unified source-instruction-edit format, so that diverse tasks can be learned jointly. Pretrained multimodal editors then learn all tasks through one editing interface and a common mask-recovery rule, without dataset-specific decoders.

This covers many tasks without sacrificing accuracy. Our single model outperforms a shared nnU-Net with half its cross-dataset variability, and generalizes better to unseen datasets. The approach also holds across editing backbones, and stronger editors yield better segmentation, so it can benefit from future progress in general-purpose models.

To complement the direct-mask generation ablation with comparisons against discriminative segmentation methods, we train a shared nnU-Net v2-based 2D model using PlainConvUNet and fine-tune a shared MedSAM model with ViT-B and its official CLIP-based text-prompt extension. MedSAM updates the text projection and mask decoder while keeping the image and text encoders frozen. Both models are trained for 20 epochs.

FLUX 3 Image Klein Base is a 5.6B image generation and editing transformer model. Comparison with discriminative segmentation baselines:

ModelAverage Dice (%)Average IoU (%)
nnU-Net74.4963.75
MedSAM72.7161.98
FLUX80.5170.68
Ours86.8378.97

W4 Computational cost and Backbone generality

ModelMean inference latency (s/image)Throughput (images/s)
nnU-Net v20.1457.61859
MedSAM0.1229.33648
FLUX 322.9800.04549
Ours48.8540.02015

For all models, we measured latency at batch size 1 and throughput at batch size 8. Generative models are slower due to iterative sampling, which is acceptable for offline analysis and quantitative research, where accuracy matters more than real-time speed.

Our approach is applicable to different multimodal editing models and is not restricted to a 20B backbone. To compare with discriminative segmentation baselines, we additionally adapt the smaller FLUX 3 model (5.6B) using the same editing formulation for 20 epochs, matching the baselines' epoch count. These results support the applicability of our paradigm across different pretrained editing backbones.

Dice/IoU (%):

DatasetnnU-NetFLUX
BBBC01083.71/72.2189.18/80.64
BriFiSeg81.42/69.0287.34/77.74
CellNuclei77.58/66.5978.47/66.38
CHASE_DB141.14/26.9573.30/57.94
CystoidFluid83.62/73.0077.71/66.20
DeepBacs68.74/52.7270.00/57.50
DRIVE45.57/30.6078.46/65.05
DynamicNuclearNet92.99/87.3991.69/85.17
FIVES70.06/55.9483.43/72.87
G102075.80/63.6570.62/58.76
GlaS64.78/50.0181.21/70.84
IDRiD91.88/86.3493.91/89.11
ISIC201880.40/70.1184.49/75.68
Kvasir-SEG87.38/80.4578.17/71.65
MoNuSAC64.28/49.2168.31/59.74
NeoPolyp63.60/52.7775.43/63.55
NuSeT93.79/88.6992.41/86.63
OCT-CME58.19/46.7975.99/65.91
PanNuke75.96/63.9764.86/53.03
PolypGen64.32/52.6172.30/61.89
REFUGE81.65/70.5876.32/62.67
RoboTool80.22/69.1586.56/79.23
STARE30.06/18.8278.88/65.25
TNBCNuclei73.45/58.6371.75/56.41
UWaterlooSkinCancer91.57/85.4083.66/78.13
WBC96.11/92.6594.72/90.41
YeaZ92.88/86.9494.57/89.85
Average74.49/63.7580.51/70.68

W5 Generalization to unseen datasets.

We directly evaluate our model, nnU-Net, and MedSAM on three external datasets excluded from task-specific training. All models are evaluated without additional fine-tuning.

  • PH²: 200 dermoscopic images from Hospital Pedro Hispano, Portugal.
  • CVC-ClinicDB: 612 colonoscopy images from Hospital Clínic de Barcelona, Spain.
  • NuInsSeg: 665 H&E histopathology patches from human and mouse tissues, Medical University of Vienna, Austria.

These results show that our model achieves competitive generalization performance on unseen datasets.

Dice/IoU (%):

DatasetnnU-NetMedSAMOurs
PH²89.67/82.5891.04/84.9291.65/84.71
CVC-ClinicDB56.62/45.9779.32/71.1483.66/76.98
NuInsSeg68.62/54.0353.10/38.9470.52/56.57
Average71.63/60.8674.48/65.0081.94/72.75

Thank you for the response and additional experiments. I acknowledge the contribution of the dataset. However, given that this is a main track submission rather than a Datasets and Benchmarks submission, I still need to consider the methodological contribution. Training nnU-Net for only 20 epochs seems insufficient (the default of nnU-Net v2 is 1,000 epochs). Can the authors confirm that the model has converged? While the relatively poor performance of MedSAM on thin vascular structures is understandable, the reported nnU-Net results are surprisingly low, particularly on STARE, DRIVE, and CHASE_DB1. More fundamentally, I remain unconvinced of the need to formulate segmentation as image editing when existing discriminative models already support unified segmentation at a substantially lower computational cost. Besides, I find the extremely high inference latency difficult to accept. Also, given that the dataset is one of the main contributions, will the authors publicly release OptiMedEdit-37K, including the editing instructions and data construction pipeline? If not, this would be another concern for me.

Thank you for your prompt feedback.

We will publicly release the dataset, the editing instructions, and the details of the data construction pipeline.

Regarding your core question: why formulate segmentation as "image editing"?

Our method performs text-instruction-guided segmentation. In medical practice, specifying the segmentation target with a text instruction is a more natural form of interaction. Clinicians often describe the structures they want to analyze in language, whereas a fixed-label model must be retrained to add or switch targets. With text instructions specifying targets on demand, one model can serve many needs and integrates readily into multimodal medical applications.

For instruction-guided segmentation, general-purpose multimodal generative models are better suited than conventional text-prompted discriminative segmenters. The latter typically condition a mask decoder on category embeddings from a separate text encoder, so language and image are fused shallowly and often within a predefined vocabulary. The former learn language understanding and image generation jointly from massive image–text data: the instruction and image are encoded together by a vision-language model and interact with image features in every layer of the generator, so more flexible instructions can be understood and executed directly in pixel space.

Building on this, prior work shows that formulating segmentation as localized editing of the source image content effectively exploits this capability (InstructDiffusion, Geng et al., 2024; Emu Edit, Sheynin et al., 2024; Qwen-Image, Wu et al., 2025). This formulation fundamentally changes how the generative model learns and anchors spatial segmentation, and it is well suited to medical imaging. Compared with asking the same model to generate masks, an edit stays within the model's pretraining distribution and anchors the target on the source anatomy, so its priors are reused rather than overwritten. Our experiments support this: the editing model outperforms instruction-guided discriminative baselines, including text-prompted MedSAM fine-tuned on the same data. nnU-Net therefore serves as a reference for segmentation accuracy rather than the paradigm we aim to replace, and it remains the more efficient choice when inference latency is critical.

Initially, to compare all models under the same number of epochs, we fixed training at 20 epochs. Following your suggestion, we increased nnU-Net training to 500 epochs. Based on validation performance, we found that 400 epochs are sufficient for the model to converge, and excessive training also reduced nnU-Net's generalization to unseen datasets. We selected the checkpoint with the best validation performance (epoch 320) and evaluated it on the test set, where it achieves an average Dice/IoU of 82.60/72.99.

EpochDice (%)IoU (%)
4075.6466.03
8078.1767.79
12079.5769.61
16079.7269.73
20080.7671.03
24081.1172.32
28080.0070.73
32082.8273.18
36080.4671.72
40081.2872.68
EpochCVC-ClinicDBHRFNuInsSegCoNSeP
10073.35% / 64.38%77.25% / 63.15%60.89% / 47.83%66.03% / 49.60%
20061.17% / 51.70%72.49% / 56.98%59.04% / 46.40%60.55% / 44.19%
24076.45% / 68.19%75.59% / 60.92%47.43% / 35.71%65.02% / 48.67%
28071.03% / 62.17%74.65% / 59.74%43.38% / 31.92%70.04% / 54.25%
32062.38% / 53.80%78.61% / 64.99%47.95% / 36.26%68.92% / 52.95%
Reviewer_VcRiOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper trains an image editing model to segment by drawing a colored overlay. A fine-tuned Qwen-Image-Edit draws the requested structure in red, green or blue at opacity 0.6. A per-pixel inversion of the blend recovers the mask. The authors convert 29 public datasets into 37,059 image, instruction and overlay records. This corpus, OptiMedEdit-37K, covers seven optical modalities.

Training has three stages. Stage I trains only the connector between the frozen encoder and the transformer. Stage II trains the full transformer. Stage III freezes it and trains GRADE, a vessel-only low-rank adapter. It adds region and centerline losses on a soft mask recovered from a one-step estimate of the edit.

The resulting model, UniMedOpti, reaches 86.17 Dice and 80.54 intersection over union (IoU) on the 7,332 test records (Figure 4). Its Dice is above the transcribed reference on 10 of 13 Table 1 rows. On FIVES, the main GRADE benchmark, it is 2.97 Dice below. The core idea is sound. Two points need checking: train and test separation, and the noise in masks recovered from generated images. Both can be checked in the rebuttal.

Critical strengths
  1. GRADE is tested against matched controls. All Stage III variants share the frozen Stage II checkpoint, the adapter capacity, the 651 vessel training records and the five-epoch schedule (Table C.3). The Flow-only variant trains the same adapter with the generative loss alone. Against it, GRADE gains 5.07 Dice on FIVES (84.06 to 89.13) and 6.30 on DRIVE (76.28 to 82.58) (Figure 5a, b). Adding the centerline boundary Dice (cbDice) loss raises FIVES centerline Dice (clDice) from 91.00 to 95.73 (Figure 6c). On a second backbone, DeepGen 1.0, GRADE adds 6.17 and 6.16 Dice over the adapted model (Figure 5c). GRADE is the paper's one new mechanism and its best supported result.

  2. The UniMedOpti scoring is fully reported. Scores are averaged per record (Appendix D.4). The threshold is fixed before test evaluation (page 4, line 193). Each transcribed reference names its source table (Table D.1). Table 1 bolds the reference wherever UniMedOpti loses. These choices make the numbers checkable.

Critical weaknesses
  1. The corpus splits lack a duplicate check across datasets and a mapping to official test sets. The paper offers OptiMedEdit-37K as a reusable dataset (page 2, lines 81 to 82). Training combines all 29 datasets. An image shared by two datasets can appear in both the training and test sets. I could not find a cross-dataset image check or a unique image count.

ISIC (International Skin Imaging Collaboration) 2016 and 2018 overlap. Mirikharaji et al. (2023) report 1,940 images shared by at least two of the 2016 to 2018 sets. Any ISIC 2016 test image among the 2,694 ISIC 2018 training records is seen in training. BKAI-IGH (800 and 200 records) and NeoPolyp (1,640 and 410) appear to come from the same 1,000 images. REFUGE has 400 test records over two targets (Table A.1). By my reading, one record per target (page 14, lines 718 to 719) means 200 test images. Please confirm.

The REFUGE reference is GCN, a graph convolutional network (Tian et al., 2020). It reports on the official 400-image test set. Its disc region also includes the cup. The paper's disc target excludes it (Figure A1 caption, page 15, line 778). REFUGE-Disc (95.52 versus 97.76) is not a direct comparison. I suggest:

  • Run a near-duplicate image search between all training and test images.
  • Report unique image counts and split sources for all 29 datasets.
  • For REFUGE, list the test identifiers and score the whole disc.
  1. Segmentation through image generation gives a noisy, indirect mask, and a simpler output exists. Generation adds noise. The model regenerates the whole image through its decoder, and the mask is the difference between output and input (Eq. 2). Any change in the regenerated pixels enters this difference and becomes mask noise. Fine details, such as thin vessels and small nuclei, may not be regenerated exactly. The paper checks background preservation only on the constructed training targets (page 17, lines 907 to 909). It does not measure this difference on model outputs. The supervision is also indirect. Stages I and II use only the flow-matching loss on latents (Eq. 4). A segmentation loss appears only in Stage III, and only for vessels. The 7,095 non-vessel test records never receive one.

    The overlay also fails at low color contrast. Each target pixel is a 0.6 blend of the overlay color and the source pixel (Eq. 1; page 4, lines 172 to 175). Appendix B.4 states that when the source color equals the overlay color, "the image contains no information about the overlay strength at that pixel" (page 20, lines 1077 to 1079). Low contrast also biases the recovered strength toward zero. A red overlay on red endoscopic tissue is one example.

    The recent Vision Banana preprint avoids the contrast problem and the need to regenerate the source (Gabeur et al., 2026). It generates a solid color map, with one color per class or per instance, and assigns each pixel to the nearest color. The output does not depend on the source color, and the source need not be regenerated. This output format could strengthen the method. I suggest:

  • On test outputs, report the output and input difference outside the target, and the false positives it causes.
  • Report the share of target pixels with low color contrast to the overlay, and Dice on those pixels.
  • Compare with solid color-map outputs under the same backbone, data and schedule.
  • Add the segmentation loss for all targets, or explain why only vessels receive it.
  • Discuss Vision Banana and explain when a blended overlay is preferable.
Questions for authors
  1. Which split selected the epoch counts and the loss weights? I could not find a validation split.
  2. Edit strength is the recovered overlay opacity, 0.6 on an ideal target. During training, GRADE's soft mask reaches 0.5 at edit strength 0.3 (Appendix B.4, Table C.1). At test time, the mask keeps pixels with strength above 0.5 (Appendix D.1). On stored outputs at thresholds 0.3 and 0.4, does GRADE keep its FIVES and DRIVE gains over the Flow-only variant (5.07 and 6.30 Dice)? A smaller gap would mean the gain comes from stronger overlay contrast, not better vessel shape.
  3. Figure 4b is the main evidence for editing (20.63 Dice over direct-mask generation) and for staged training (6.54 over non-staged editing). Which parameters, losses and instructions does each baseline use in each phase (Appendix C.5)? How is a decoded RGB mask made binary? All variants end with 5 vessel-only epochs (page 26, lines 1352 to 1354). Vessels are only 237 of 7,332 test records, so at least 3.3 of the 6.54 points come from non-vessel records. Please report per-dataset Dice, and both baselines after epoch 17, before the vessel-only phase. Also give the inference steps, guidance scale and seed (Appendix C.4).
1 comment

Thank you for your thorough and constructive review.

W1. Deduplication, splits, and REFUGE.

Our original duplicate checks were insufficient. After screening all datasets for near-duplicates, we removed ISIC2016 and BKAI-IGH, 543 CellNuclei images shared with NuSeT (127 retained), and 988 OCT-CME images shared with CystoidFluid (472 retained). No further overlaps were identified. We also corrected BriFiSeg's record counts. The corpus contains 27 datasets and 34,369 image–target records (splits in Q3). REFUGE has 400 official test images and 800 records (T0001–T0400), previously misreported as 400 records. Our disc target excludes the cup, so we withdraw the comparison with GCN's whole-disc score.

W2–W4. Background noise, low contrast, and supervision.

On 6,864 test records, the pooled RGB MAE between the generated edit and input on GT background pixels was 0.0099 on [0,1]. Background FPR was 1.0764% at the mask threshold τ=0.5.

When source and overlay colors coincide, the pixel carries no information about overlay strength. We grouped pixels by their source-to-overlay RGB distance, using 0.4 as the cutoff.

RGB distanceShare of GT foregroundPooled DiceRecall
< 0.40.7841%96.23%93.94%
≥ 0.499.2159%90.20%87.42%

Stage III uses cbDice to address breaks in thin vessel branches. A zero residual gate preserves the Stage II mapping for non-vessel prompts (Eq. A2).

Vision Banana's solid maps avoid source-color ambiguity. Our 0.6-opacity RGB overlays preserve source texture through the editor's native interface. Color defines the direction for mask recovery and differentiable geometric supervision. Only 0.7841% of GT foreground pixels have source-to-overlay RGB distance <0.4, with 96.23% Dice. Overlays outperform direct masks by 20.63 Dice points under matched settings (Fig. 4b).

Q1–Q2. Data splits and threshold sensitivity.

We prespecified the stage lengths and GRADE loss weights (Table C.1). To select checkpoints in the rerun, we used official or benchmark validation splits where available and otherwise held out 10% of the training data.

We rescored stored outputs from the two Q3 models. Cells give non-staged (flow-only) / staged (GRADE) Dice (%).

Datasetτ = 0.3τ = 0.4τ = 0.5
FIVES84.03/88.1983.95/88.4783.84/88.01
DRIVE78.12/83.6077.84/83.1177.13/82.89

Lowering τ from 0.5 to 0.3 leaves the gaps nearly unchanged on FIVES (4.17 to 4.16 points) and DRIVE (5.76 to 5.48). These gains compare the full recipes.

Q3. Staged-training comparison.

Both models used target-and-color instructions and 20 mixed-dataset epochs of flow matching. Staged training updated the connector for 3 epochs and the full backbone for 17. Non-staged training updated the full backbone for all 20. Both then froze it for 5 vessel-only LoRA epochs. The non-staged model continued flow matching. GRADE added BCE, soft Dice, and cbDice. Inference used 40 steps, CFG 4.0, and seed 42.

Of the 4.41-point gain in dataset-macro Dice, 3.91 points came from the 23 non-vessel datasets. The table gives record counts and final test Dice/IoU (%). Vessel adaptation began after epoch 20 here.

DatasetTrain/Val/TestNon-stagedStaged
BBBC01070/10/2085.90/77.0588.34/79.29
BriFiSeg1005/115/24086.03/80.2391.44/84.32
CellNuclei90/11/2681.70/70.9287.59/78.17
CHASE_DB119/2/772.78/57.2973.57/60.16
CystoidFluid703/101/20277.13/66.2587.94/78.16
DeepBacs17/2/1585.35/76.0887.07/77.16
DRIVE18/2/2077.13/63.6782.89/72.77
DynamicNuclearNet4950/1417/71792.50/88.2093.54/88.22
FIVES540/60/20083.84/73.0488.01/80.85
G10201304/144/36283.61/75.1286.01/76.71
GlaS76/9/8084.85/77.6792.14/86.61
IDRiD47/6/2788.93/82.9295.97/92.50
ISIC20182594/100/100085.39/78.7292.58/86.20
Kvasir-SEG700/100/20086.69/81.6291.03/86.14
MoNuSAC188/21/10172.15/59.0173.63/59.75
NeoPolyp1478/162/41081.43/76.9185.88/80.76
NuSeT2385/340/68391.51/86.5993.39/87.98
OCT-CME333/37/10280.24/70.0982.29/72.33
PanNuke3824/425/85864.00/52.5264.20/51.65
PolypGen1001/111/29977.30/71.0882.36/75.33
REFUGE800/800/80080.67/70.2094.84/92.11
RoboTool350/50/10088.92/83.4992.41/87.11
STARE9/1/1072.14/57.0175.01/60.45
TNBCNuclei35/5/1074.15/60.5175.81/61.44
UWaterlooSkinCancer143/19/4492.11/87.7193.06/88.11
WBC280/40/8087.51/83.7096.23/92.98
YeaZ360/96/25191.32/87.3597.16/94.86
Total / Macro avg.23319/4186/686482.42/73.8986.83/78.97

The direct-mask baseline uses binary GT masks. We resize the VAE-decoded RGB output to the source image size and label pixels with mean RGB intensity >0.5 on [0,1] as foreground.

Reviewer_MQRROpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a unified paradigm called Segmentation as Editing, aimed at addressing two major challenges in medical optical image segmentation: the lack of a unified multi-modal segmentation dataset, and the absence of a single segmentation model that performs well across diverse optical imaging modalities. Its main contributions include the OptiMedEdit-37K​ dataset, the UniMedOpti​ unified segmentation model, and the geometry-aware residual adaptation method GRADE. Overall, the paper successfully transforms medical image segmentation into an image editing problem, and through carefully designed datasets and a staged training strategy, achieves a unified segmentation model that combines cross-modal generality with high accuracy.

Critical strengths
  1. The problem addressed in the paper has practical significance, and the data integration work holds certain value.
  2. The methodology is logically coherent, and the key mathematical relationships are essentially valid.
  3. The vascular geometry supervision is supported by reasonably credible experimental evidence.
Critical weaknesses
  1. Fig. 4a compares general-purpose editing models without fine-tuning on this dataset against the fully trained proposed model. This result demonstrates the effectiveness of task adaptation, but cannot alone prove the unique value of the dataset or the superiority of the method. It is recommended to at least supplement with a text-prompted unified segmentation baseline, such as BiomedParse, as well as a discriminative segmentation baseline trained on the same mixed dataset.
  2. There is a lack of generalization evidence on unseen data. Current experiments can support that the model covers multi-modal tasks, but they are insufficient to demonstrate that it can generalize to unseen datasets, hospitals, or imaging conditions.
  3. The authors need to provide failure cases and analyze under what circumstances the model fails.
  4. I believe the authors need to explain what unique advantages there are in imposing geometric supervision through the editing output.
Questions for authors
  1. Fig. 4a compares general-purpose editing models without fine-tuning on this dataset against the fully trained proposed model. This result demonstrates the effectiveness of task adaptation, but cannot alone prove the unique value of the dataset or the superiority of the method. It is recommended to at least supplement with a text-prompted unified segmentation baseline, such as BiomedParse, as well as a discriminative segmentation baseline trained on the same mixed dataset.
  2. There is a lack of generalization evidence on unseen data. Current experiments can support that the model covers multi-modal tasks, but they are insufficient to demonstrate that it can generalize to unseen datasets, hospitals, or imaging conditions.
  3. The authors need to provide failure cases and analyze under what circumstances the model fails.
  4. I believe the authors need to explain what unique advantages there are in imposing geometric supervision through the editing output.
3 comments

Regarding the questions I raised, no response has been provided to date. Receiving clarification on these points before I proceed would be highly beneficial for my evaluation.

Thank you for your careful review and constructive suggestions. We address your four comments below.

1. Additional text-prompted unified and discriminative segmentation baselines.

We re-curated the corpus by searching all training and test images for near-duplicates, removing the overlapping ISIC 2016 and BKAI-IGH, and re-splitting the 27 remaining datasets (34,369 records). All new results use this corpus. We evaluate three publicly available text-prompted segmentation models directly on our test set.

  • BiomedParse v1: Microsoft's original 2D model (biomedparse_v1.pt).
  • Medical SAM3: AIM-Research-Lab's 2D model (checkpoint_2D.pt).
  • MedSAM3-v1: Liu et al.'s model using best_lora_weights.pt with the original SAM 3 backbone.

Their performance is summarized below. This result further demonstrates the effectiveness of task adaptation.

Dice/IoU (%):

ModelAverage DiceAverage IoU
BiomedParse v150.0843.89
Medical SAM3 (2D)54.6043.44
MedSAM3-v142.3435.73
Ours86.8378.97

We additionally train a single shared nnU-Net v2-based 2D model and fine-tune a single shared text-promptable MedSAM on the combined datasets. The nnU-Net baseline uses the official PlainConvUNet architecture. MedSAM uses the original ViT-B backbone and its official CLIP-based text-prompt extension. Fine-tuning updates the text projection and mask decoder with frozen image and text encoders.

Both models are trained for 20 epochs. Results are computed at the native ground-truth resolution. These results further demonstrate the superiority of our method.

ModelAverage DiceAverage IoU
nnU-Net74.4963.75
MedSAM72.7161.98
Ours86.8378.97

2. Additional evidence of generalization to unseen datasets.

To assess generalization to unseen datasets, we evaluate our model on three external datasets: PH², CVC-ClinicDB, and NuInsSeg. These datasets were excluded from task-specific training and share no images with ours. We apply the trained model directly to these datasets without additional fine-tuning.

  • PH²: 200 dermoscopic images from Hospital Pedro Hispano, Portugal.
  • CVC-ClinicDB: 612 colonoscopy images from Hospital Clínic de Barcelona, Spain.
  • NuInsSeg: 665 H&E histopathology patches from human and mouse tissues, Medical University of Vienna, Austria.

We use these three external datasets to evaluate our model's generalization ability and compare its performance with that of nnU-Net and MedSAM. These results show that our model achieves competitive generalization performance on unseen datasets.

DatasetnnU-Net DicennU-Net IoUMedSAM DiceMedSAM IoUOurs DiceOurs IoU
PH²89.6782.5891.0484.9291.6584.71
CVC-ClinicDB56.6245.9779.3271.1483.6676.98
NuInsSeg68.6254.0353.1038.9470.5256.57
Average71.6360.8674.4865.0081.9472.75

3. Failure case analysis.

Our model may fail under particularly challenging conditions, such as severe blur, extremely low target-background or target-overlay contrast, or very small and densely packed structures, whose fine details may be lost during VAE compression. Examples: (i) Dense, multi-class cells: on PanNuke (64.20 vs. 75.96 Dice for nnU-Net), clustered nuclei can merge after VAE downsampling, and similar cell types are hard to separate by instruction alone. (ii) Low overlay contrast: when the source and overlay colors are close, the recovery margin shrinks with ∥c−xi∥\lVert c - x_i\rVert, so small generation errors cause under-segmentation (App. B.4). We will add failure-case visualizations to Appendix E.

4. Advantages of geometric supervision through the editing output. Compared with a mask head or latent-only supervision, applying geometric losses to masks recovered from the generated edit has three advantages:

  • Unified interface. Recovery is analytic (Eq. 2), so no segmentation head is needed and the signal must improve the edit itself (Eq. 8). Its gradient lies along the edit direction (Eq. A9), leaving the source appearance untouched.
  • Pixel-space topology in one forward pass. Flow matching does not explicitly reward thin-vessel connectivity. GRADE decodes the one-step clean estimate (Eq. 6) through the frozen VAE and applies BCE, Dice, and cbDice in pixel space with a single MMDiT evaluation, without unrolled sampling.
  • Target-gated modularity. Gated residuals refine vessel targets while every other target keeps exactly its Stage-II function (Eq. A2), preserving the unified model's balanced performance.

At matched capacity and schedule, GRADE improves Dice by 5.07/6.30 over flow-only residuals on FIVES/DRIVE (Fig. 5), cbDice adds 4.73 clDice (Fig. 6), and the gain transfers to DeepGen (+6.2 Dice).

We thank the authors for their response and the efforts made on the experiments, which have largely addressed my concerns. At the same time, this has also given me a more comprehensive understanding of this paper. Therefore, I have raised my confidence score from 2 to 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.