- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes a pipeline for generating image/text editing pairs for multi-instance editing, including attribute-based addition or removal of objects. Starting from object proposal, the pipeline automatically expands it to a an image generation prompts with several variations of object count and attributes. Once the images are generated, Gemini VLM and SAM 3 are iteratively used to validate that the correct count in the generated images. Finally, a number of objects are inpainted to simulate removal edit, and are verified based on the same VLM and SAM 3 loop. To construct addition edits, the verified removal edits are reversed. Experiments show that both open-source and proprietary models struggle on the proposed benchmark. By finetuning FLUX Kontext on MOMI training set, the performance improves but still achieves a low accuracy of 16% revealing a huge gap on text-based multi-instance editing.
Critical strengths
-
The paper includes extensive details about the pipeline that can help reproduce the pipeline.
-
The experiments section provide a thorough analysis of the results across different models and types of edits.
Critical weaknesses
-
My major concern about this benchmark is either this setup of using text for multi-instance image editing is actually realistic. When a human wants to edit an image with many objects, referring to objects using the complex language depicted in Figure 1 (e.g. lower-left, mid-left, center, mid-right) can be challenging. A more natural way to do this task would be to point to a specific object and describe the edit similar to how GPT-Image performs editing now. Therefore, the paper needs a convincing motivation on how this text-based approach can be usable.
-
Using inpainting to perform removal edits is not optimal as it does not edit the interaction between the object and the environment (e.g. reflection, occlusion, ..etc). Also inpainting is not guaranteed to always remove the object rather that re-generating it fully or partially based on the silhouette of the inpainting mask. This creates a domain gap to real image editing and raises concerns about the realism and accuracy of the generated data.
-
Using CLIP to automatically detect failed edits is not reliable. CLIP provides a global embedding that only changes if the global semantics of the scene were changed. For inpainting artifacts or filed edits, CLIP is unlikely to capture them as demonstrated in Cvejic, Aleksandar, et al. "NearID: Identity Representation Learning via Near-identity Distractors." ECCV, 2026.
-
For compound edits, the same image is fed to the diffusion models several times which can cause the visual quality of the image to degraded due to the several VAE encoding/decoding iterations.
Minor:
- The abbreviations SOMI and MOMI are used at line 76 before getting defined.
Questions for authors
-
How is the taxonomy of the dataset defined (i.e. what objects to include)?
-
Why are two different image generation models used to generate SOMI and MOMI images as per line 104?
-
Given that some images have a large number of objects, how many iterations on average needed for the image generation model to produce a correct image?
-
How would any of the image editing models perform if equipped with the validation in section 2.2 to iteratively do the edit until it converges to the correct edit?