- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents MieDB-100k, a dataset of 104,267 text-guided editing triplets built from 48 public datasets across 10 imaging modalities. Perception tasks paint segmentation labels onto the image as color overlays. Modification tasks use modality-specific FLUX inpainting experts to turn lesion-bearing images into healthy counterparts, followed by automatic filtering and expert review. Transformation tasks use paired images from public repositories. OmniGen2 finetuned on the training split achieves the best results in Table 2 across all three perspectives.
Critical strengths
The Modification pipeline is a reasonable way to obtain counterfactual pairs with consistent backgrounds. Inpainting is confined to the lesion mask, and expert review after automatic filtering discarded 9,294 of 25,677 candidate pairs. The Turing test in Table 16 gives direct evidence on the realism of the inpainted tissue.
Critical weaknesses
- The benchmark is built from the test splits of the same source datasets and follows the same task construction used for training. OmniGen2-MIE is therefore evaluated on a distribution closely matched to its training data, while the other models are used zero-shot. For lesion removal, the Rubric-S reference is also generated by the FLUX-based inpainting pipeline. In Figure 5, Nano Banana Pro scores 58.1 on the unseen bone metastasis task, compared with 52.2 for OmniGen2-MIE. Table 15 only compares OmniGen2-MIE with OmniGen2, so the evidence for outperforming proprietary models mainly comes from the MieDB-100k benchmark.
- The paper presents MieDB-100k as a higher-quality alternative to existing medical editing datasets, but no model is trained on an existing dataset. The effect of the proposed curation is therefore difficult to separate from general in-domain finetuning.
- Section 4.2 links low Perception scores to a lack of medical knowledge. However, Table 3 lists a Perception ACC of 0.000 for the M-only model while its Modification RubricScore is 57.5. Low overlay performance may also reflect difficulty with the required output format.
Questions for authors
- Section 4.2 treats low Perception scores as evidence of missing medical knowledge. How do the authors reconcile this with the M-only model in Table 3, which has a Perception ACC of 0.000 but a Modification RubricScore of 57.5?
- The reported correlation of −0.905 appears to match the correlation between the ten model-level averages in Table 2. How many cases were ranked by the three evaluators, and how well do Rubric-S and human rankings agree at the case level?
4 comments
Reply to Reviewer CKiS (2/2)
Reconciling Low Perception Scores of Zero-Shot Baselines vs. the M-only Model (Q1 & W3)
Thank you for raising this insightful point. The low Perception scores of zero-shot general-purpose baselines and our single-task M-only model arise from two fundamentally different mechanisms:
- Zero-Shot Baselines Fail Due to Missing Radiological Knowledge, Not Format Unfamiliarity: Untouched baselines already possess the instruction-following ability to paint semi-transparent colored overlays. As shown in our modality-wise breakdown (Appendix E.1, Fig. 10), under the exact same prompt format, baselines achieve strong DICE scores (~0.60–0.80) on optical modalities resembling natural images, where OmniGen2 scores 0.248 overall. Furthermore, our least-squares alpha de-blending extraction (Appendix D.1, Eq. 2) is invariant to opacity variations. Their failure specifically on non-optical scans (CT, MRI, OCT) occurs because they paint the requested color onto the wrong anatomical structures (Fig. 4), exposing a lack of radiological domain knowledge.
- M-only Scores 0.000 Due to Single-Task Mode Collapse (Format Forgetting): Unlike zero-shot baselines, M-only is full-parameter finetuned for 20,000 steps exclusively on 41,644 Modification triplets, where every target output is a natural-looking clinical scan without any artificial color overlays. Consequently, M-only experiences catastrophic forgetting of the color-painting output mode.
These support the core of our work: "Perception Matters for Modification". While M-only learns implicit localization for trained lesion types, explicitly retaining and injecting anatomical understanding via joint training can further boost Modification Rubric-S from 57.5 to 65.9, and enables zero-shot concept transfer to unseen lesions (Fig. 5).
Case-Level Agreement and Sample Size for Human Preference Ranking (Q2)
In Sec. 3.3.2, the reported ρ=−0.905 indeed measures the model-level Spearman rank correlation across the 10 benchmarked models. Here we clarify the evaluation scale and case-level agreement:
- Number of Ranked Cases: All three clinical evaluators independently ranked all 618 test cases in the Modification benchmark (comparing all 10 model outputs per case, totaling 6180 generated images evaluated per clinician).
- Case-Level Correlation: When computing the Spearman rank correlation between VLM Rubric-S and each of the three clinical evaluators' preference rankings within each individual test case and averaging across all cases, we obtain a case-level correlation ρ of -0.687, -0.700, -0.673 ( with all p<1×10−3). This confirms high case-level agreement between our rubric-guided VLM evaluator and individual clinical experts.
We hope these analyses and additional experiments fully address your concerns.
Thanks for the response and the extra experiments. This addresses several of my initial questions, though I still have two quick follow-ups before finalizing my score. You noted that OmniGen2-MIE still beats Nano Banana Pro on the Lesion Addition subset with real targets. Could you post the exact Rubric-S and Pref-Rank numbers broken down into Lesion Addition vs. Lesion Removal? Also, the Med-banana-50K comparison is still tested on MieDB-100k, and Table 15 only compares against untuned OmniGen2. Could you run OmniGen2-MedBanana and at least one strong baseline such as Nano Banana Pro or Qwen-Image-Edit on MedE-Bench?
Reply to Reviewer CKiS (Round II)
Thank you for your prompt reply and constructive follow-up questions. Here we supplement the detailed breakdown:
1. Lesion Addition vs. Lesion Removal
| Model | Lesion Addition (100% Real Target ) Rubric-S () / Pref-Rank () | Lesion Removal (Inpainted Target ) Rubric-S () / Pref-Rank () |
|---|---|---|
| Nano Banana Pro | 67.78 / 2.03 | 58.94 / 2.05 |
| OmniGen2-MIE (Ours) | 69.09 / 1.38 | 62.53 / 1.40 |
OmniGen2-MIE consistently outperforms Nano Banana Pro across both directions. Specifically, on the Lesion Addition subset (the ground-truth targets () are 100% real clinical scans), OmniGen2-MIE still achieves a clear lead, confirming that its superiority holds on authentic clinical pathology targets rather than merely matching FLUX-inpainted references.
2. Expanded External Evaluation on MedE-Bench
| Model | Size | CLIP () | PSNR () | SSIM () | EA () | CP () | VQ () |
|---|---|---|---|---|---|---|---|
| OmniGen2 | 7B | 0.234 | 14.6 | 0.898 | 0.35 | 0.64 | 0.51 |
| Qwen-Image-Edit | 27B | 0.263 | 15.6 | 0.922 | 0.47 | 0.66 | 0.52 |
| OmniGen2-MedBanana | 7B | 0.261 | 15.0 | 0.935 | 0.42 | 0.68 | 0.54 |
| OmniGen2-MIE (Ours) | 7B | 0.267 | 15.4 | 0.951 | 0.46 | 0.77 | 0.57 |
We note that because the data distribution and longitudinal collection setup of MedE-Bench differ from our dataset, the performance margins are naturally more moderate than those measured on the in-distribution benchmark. Nevertheless, two clear conclusions emerge:
- Direct Dataset Comparison: Evaluated on an independent, real-pair external benchmark using the exact same 7B backbone, OmniGen2-MIE outperforms OmniGen2-MedBanana across all six metrics, directly validating the superior quality and generalizability of our curation pipeline over Med-banana-50K.
- Comparison with Strong Baselines: Despite having nearly fewer parameters than Qwen-Image-Edit (7B vs. 27B), OmniGen2-MIE achieves the best overall performance.
We will incorporate both tables into the revised manuscript and thank you again for your constructive engagement!
Reply to Reviewer CKiS (1/2)
We sincerely thank the reviewer for the thorough review and constructive feedback, and for recognizing the soundness of our Modification pipeline, rigorous expert filtering, and clinical Turing test validation. Below, we address your questions and concerns point by point.
Benchmark Distribution, Ground-Truth Authenticity, and External Comparisons (W1)
We will refine our wording in the Abstract and Introduction to state more precisely that OmniGen2-MIE "surpasses open-source and proprietary models on the MieDB-100k benchmark while achieving highly competitive zero-shot generalization approaching SOTA proprietary models on unseen editing targets". We address the rest concern regarding in-distribution evaluation and FLUX-generated references from three perspectives:
Isolating the Effect of Our Curation Pipeline from General In-Domain Finetuning (W2)
To directly separate the effect of our proposed curation pipeline from generic in-domain finetuning, we finetuned the exact same OmniGen2 backbone on Med-banana-50K using the same training recipe, and compared it against our models:
This controlled comparison attribute the performance gains into three distinct factors:
We will include this OmniGen2-MedBanana comparison table and analysis in the revised manuscript.