- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Segmentation as Editing, a framework that formulates medical image segmentation as instruction-guided image editing. The authors construct OptiMedEdit-37K across seven optical imaging modalities and fine-tune Qwen-Image-Edit for unified segmentation. They further introduce GRADE, which combines LoRA with geometric supervision to improve vessel segmentation.
Critical strengths
The experiments cover diverse imaging modalities and segmentation tasks and show consistent improvements from geometric supervision
Critical weaknesses
-
The novelty is somewhat limited, and I don't find the idea particularly interesting.
-
The paper lacks comparisons with conventional segmentation models such as nnU-Net, and SAM/MedSAM.
-
The direct-mask baseline still uses a generative model. It remains unclear whether image editing offers advantages over conventional discriminative segmentation.
-
The method relies on a 20B generative model, but training costs, inference latency, and throughput are not reported or compared with segmentation baselines.
-
Most evaluations use held-out samples from datasets included in training. Experiments on unseen datasets or modalities would better support the claim of unified segmentation.
Questions for authors
see weakness
4 comments
Thank you for your time and valuable feedback.
W1-W3 Motivation, novelty, and comparison with discriminative segmentation.
Our goal is a single reusable segmentation model for heterogeneous medical optical imaging. As modalities and target structures multiply, maintaining a separate model for each task becomes increasingly burdensome, and annotations differ in definition and format across datasets.
We address data and model adaptation together. Our curated corpus organizes the existing image-mask annotations scattered across datasets into a unified source-instruction-edit format, so that diverse tasks can be learned jointly. Pretrained multimodal editors then learn all tasks through one editing interface and a common mask-recovery rule, without dataset-specific decoders.
This covers many tasks without sacrificing accuracy. Our single model outperforms a shared nnU-Net with half its cross-dataset variability, and generalizes better to unseen datasets. The approach also holds across editing backbones, and stronger editors yield better segmentation, so it can benefit from future progress in general-purpose models.
To complement the direct-mask generation ablation with comparisons against discriminative segmentation methods, we train a shared nnU-Net v2-based 2D model using PlainConvUNet and fine-tune a shared MedSAM model with ViT-B and its official CLIP-based text-prompt extension. MedSAM updates the text projection and mask decoder while keeping the image and text encoders frozen. Both models are trained for 20 epochs.
FLUX 3 Image Klein Base is a 5.6B image generation and editing transformer model. Comparison with discriminative segmentation baselines:
| Model | Average Dice (%) | Average IoU (%) |
|---|---|---|
| nnU-Net | 74.49 | 63.75 |
| MedSAM | 72.71 | 61.98 |
| FLUX | 80.51 | 70.68 |
| Ours | 86.83 | 78.97 |
W4 Computational cost and Backbone generality
| Model | Mean inference latency (s/image) | Throughput (images/s) |
|---|---|---|
| nnU-Net v2 | 0.145 | 7.61859 |
| MedSAM | 0.122 | 9.33648 |
| FLUX 3 | 22.980 | 0.04549 |
| Ours | 48.854 | 0.02015 |
For all models, we measured latency at batch size 1 and throughput at batch size 8. Generative models are slower due to iterative sampling, which is acceptable for offline analysis and quantitative research, where accuracy matters more than real-time speed.
Our approach is applicable to different multimodal editing models and is not restricted to a 20B backbone. To compare with discriminative segmentation baselines, we additionally adapt the smaller FLUX 3 model (5.6B) using the same editing formulation for 20 epochs, matching the baselines' epoch count. These results support the applicability of our paradigm across different pretrained editing backbones.
Dice/IoU (%):
| Dataset | nnU-Net | FLUX |
|---|---|---|
| BBBC010 | 83.71/72.21 | 89.18/80.64 |
| BriFiSeg | 81.42/69.02 | 87.34/77.74 |
| CellNuclei | 77.58/66.59 | 78.47/66.38 |
| CHASE_DB1 | 41.14/26.95 | 73.30/57.94 |
| CystoidFluid | 83.62/73.00 | 77.71/66.20 |
| DeepBacs | 68.74/52.72 | 70.00/57.50 |
| DRIVE | 45.57/30.60 | 78.46/65.05 |
| DynamicNuclearNet | 92.99/87.39 | 91.69/85.17 |
| FIVES | 70.06/55.94 | 83.43/72.87 |
| G1020 | 75.80/63.65 | 70.62/58.76 |
| GlaS | 64.78/50.01 | 81.21/70.84 |
| IDRiD | 91.88/86.34 | 93.91/89.11 |
| ISIC2018 | 80.40/70.11 | 84.49/75.68 |
| Kvasir-SEG | 87.38/80.45 | 78.17/71.65 |
| MoNuSAC | 64.28/49.21 | 68.31/59.74 |
| NeoPolyp | 63.60/52.77 | 75.43/63.55 |
| NuSeT | 93.79/88.69 | 92.41/86.63 |
| OCT-CME | 58.19/46.79 | 75.99/65.91 |
| PanNuke | 75.96/63.97 | 64.86/53.03 |
| PolypGen | 64.32/52.61 | 72.30/61.89 |
| REFUGE | 81.65/70.58 | 76.32/62.67 |
| RoboTool | 80.22/69.15 | 86.56/79.23 |
| STARE | 30.06/18.82 | 78.88/65.25 |
| TNBCNuclei | 73.45/58.63 | 71.75/56.41 |
| UWaterlooSkinCancer | 91.57/85.40 | 83.66/78.13 |
| WBC | 96.11/92.65 | 94.72/90.41 |
| YeaZ | 92.88/86.94 | 94.57/89.85 |
| Average | 74.49/63.75 | 80.51/70.68 |
W5 Generalization to unseen datasets.
We directly evaluate our model, nnU-Net, and MedSAM on three external datasets excluded from task-specific training. All models are evaluated without additional fine-tuning.
- PH²: 200 dermoscopic images from Hospital Pedro Hispano, Portugal.
- CVC-ClinicDB: 612 colonoscopy images from Hospital Clínic de Barcelona, Spain.
- NuInsSeg: 665 H&E histopathology patches from human and mouse tissues, Medical University of Vienna, Austria.
These results show that our model achieves competitive generalization performance on unseen datasets.
Dice/IoU (%):
| Dataset | nnU-Net | MedSAM | Ours |
|---|---|---|---|
| PH² | 89.67/82.58 | 91.04/84.92 | 91.65/84.71 |
| CVC-ClinicDB | 56.62/45.97 | 79.32/71.14 | 83.66/76.98 |
| NuInsSeg | 68.62/54.03 | 53.10/38.94 | 70.52/56.57 |
| Average | 71.63/60.86 | 74.48/65.00 | 81.94/72.75 |
Thank you for the response and additional experiments. I acknowledge the contribution of the dataset. However, given that this is a main track submission rather than a Datasets and Benchmarks submission, I still need to consider the methodological contribution. Training nnU-Net for only 20 epochs seems insufficient (the default of nnU-Net v2 is 1,000 epochs). Can the authors confirm that the model has converged? While the relatively poor performance of MedSAM on thin vascular structures is understandable, the reported nnU-Net results are surprisingly low, particularly on STARE, DRIVE, and CHASE_DB1. More fundamentally, I remain unconvinced of the need to formulate segmentation as image editing when existing discriminative models already support unified segmentation at a substantially lower computational cost. Besides, I find the extremely high inference latency difficult to accept. Also, given that the dataset is one of the main contributions, will the authors publicly release OptiMedEdit-37K, including the editing instructions and data construction pipeline? If not, this would be another concern for me.
Thank you for your prompt feedback.
We will publicly release the dataset, the editing instructions, and the details of the data construction pipeline.
Regarding your core question: why formulate segmentation as "image editing"?
Our method performs text-instruction-guided segmentation. In medical practice, specifying the segmentation target with a text instruction is a more natural form of interaction. Clinicians often describe the structures they want to analyze in language, whereas a fixed-label model must be retrained to add or switch targets. With text instructions specifying targets on demand, one model can serve many needs and integrates readily into multimodal medical applications.
For instruction-guided segmentation, general-purpose multimodal generative models are better suited than conventional text-prompted discriminative segmenters. The latter typically condition a mask decoder on category embeddings from a separate text encoder, so language and image are fused shallowly and often within a predefined vocabulary. The former learn language understanding and image generation jointly from massive image–text data: the instruction and image are encoded together by a vision-language model and interact with image features in every layer of the generator, so more flexible instructions can be understood and executed directly in pixel space.
Building on this, prior work shows that formulating segmentation as localized editing of the source image content effectively exploits this capability (InstructDiffusion, Geng et al., 2024; Emu Edit, Sheynin et al., 2024; Qwen-Image, Wu et al., 2025). This formulation fundamentally changes how the generative model learns and anchors spatial segmentation, and it is well suited to medical imaging. Compared with asking the same model to generate masks, an edit stays within the model's pretraining distribution and anchors the target on the source anatomy, so its priors are reused rather than overwritten. Our experiments support this: the editing model outperforms instruction-guided discriminative baselines, including text-prompted MedSAM fine-tuned on the same data. nnU-Net therefore serves as a reference for segmentation accuracy rather than the paradigm we aim to replace, and it remains the more efficient choice when inference latency is critical.
Initially, to compare all models under the same number of epochs, we fixed training at 20 epochs. Following your suggestion, we increased nnU-Net training to 500 epochs. Based on validation performance, we found that 400 epochs are sufficient for the model to converge, and excessive training also reduced nnU-Net's generalization to unseen datasets. We selected the checkpoint with the best validation performance (epoch 320) and evaluated it on the test set, where it achieves an average Dice/IoU of 82.60/72.99.
| Epoch | Dice (%) | IoU (%) |
|---|---|---|
| 40 | 75.64 | 66.03 |
| 80 | 78.17 | 67.79 |
| 120 | 79.57 | 69.61 |
| 160 | 79.72 | 69.73 |
| 200 | 80.76 | 71.03 |
| 240 | 81.11 | 72.32 |
| 280 | 80.00 | 70.73 |
| 320 | 82.82 | 73.18 |
| 360 | 80.46 | 71.72 |
| 400 | 81.28 | 72.68 |
| Epoch | CVC-ClinicDB | HRF | NuInsSeg | CoNSeP |
|---|---|---|---|---|
| 100 | 73.35% / 64.38% | 77.25% / 63.15% | 60.89% / 47.83% | 66.03% / 49.60% |
| 200 | 61.17% / 51.70% | 72.49% / 56.98% | 59.04% / 46.40% | 60.55% / 44.19% |
| 240 | 76.45% / 68.19% | 75.59% / 60.92% | 47.43% / 35.71% | 65.02% / 48.67% |
| 280 | 71.03% / 62.17% | 74.65% / 59.74% | 43.38% / 31.92% | 70.04% / 54.25% |
| 320 | 62.38% / 53.80% | 78.61% / 64.99% | 47.95% / 36.26% | 68.92% / 52.95% |
I have not yet received or seen any response from the authors and would appreciate clarification on the questions I raised.