acceptodds
Withdrawn from ICLR 2027

PGAMIE: A PIPELINE FOR GENERATING ACCURATE MULTI-INSTANCE IMAGE EDITS

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Text-guided image editing has advanced rapidly, enabling increasingly complex image modifications through natural-language instructions. Despite this progress, models often struggle to add or remove an exact number of object instances, particularly when instructions involve multiple object categories or edit operations. We present the first systematic evaluation of multi-instance editing and introduce , an automated pipeline for constructing verified multi-instance, multi-category image-editing data. The pipeline generates count-verified scenes and constructs verified addition and removal edits through instance segmentation, inpainting, and agreement-based count verification, supporting single-category, multi-category, and compound edits in scenes containing up to 44 object instances. Using this pipeline, we create MOMI, a 20k-record training dataset and a multi-category benchmark, together with two complementary benchmarks for out-of-distribution single-category and real-image evaluation. Our evaluation reveals that multi-instance editing remains challenging for current image editors, with performance degrading sharply as requested object counts and edit complexity increase. Fine-tuning Flux Kontext on MOMI achieves the highest edit accuracy on the MOMI benchmark at 16.3%, compared with 2.0% for the base model and 14.3% for Nanobanana Pro.

Primary area:
datasets and benchmarks
Keywords:
Text-guided image editing, multi-instance image editing, object counting, instruction following, image editing benchmarks, dataset generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KNejOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a pipeline for generating image/text editing pairs for multi-instance editing, including attribute-based addition or removal of objects. Starting from object proposal, the pipeline automatically expands it to a an image generation prompts with several variations of object count and attributes. Once the images are generated, Gemini VLM and SAM 3 are iteratively used to validate that the correct count in the generated images. Finally, a number of objects are inpainted to simulate removal edit, and are verified based on the same VLM and SAM 3 loop. To construct addition edits, the verified removal edits are reversed. Experiments show that both open-source and proprietary models struggle on the proposed benchmark. By finetuning FLUX Kontext on MOMI training set, the performance improves but still achieves a low accuracy of 16% revealing a huge gap on text-based multi-instance editing.

Critical strengths
  • The paper includes extensive details about the pipeline that can help reproduce the pipeline.

  • The experiments section provide a thorough analysis of the results across different models and types of edits.

Critical weaknesses
  1. My major concern about this benchmark is either this setup of using text for multi-instance image editing is actually realistic. When a human wants to edit an image with many objects, referring to objects using the complex language depicted in Figure 1 (e.g. lower-left, mid-left, center, mid-right) can be challenging. A more natural way to do this task would be to point to a specific object and describe the edit similar to how GPT-Image performs editing now. Therefore, the paper needs a convincing motivation on how this text-based approach can be usable.

  2. Using inpainting to perform removal edits is not optimal as it does not edit the interaction between the object and the environment (e.g. reflection, occlusion, ..etc). Also inpainting is not guaranteed to always remove the object rather that re-generating it fully or partially based on the silhouette of the inpainting mask. This creates a domain gap to real image editing and raises concerns about the realism and accuracy of the generated data.

  3. Using CLIP to automatically detect failed edits is not reliable. CLIP provides a global embedding that only changes if the global semantics of the scene were changed. For inpainting artifacts or filed edits, CLIP is unlikely to capture them as demonstrated in Cvejic, Aleksandar, et al. "NearID: Identity Representation Learning via Near-identity Distractors." ECCV, 2026.

  4. For compound edits, the same image is fed to the diffusion models several times which can cause the visual quality of the image to degraded due to the several VAE encoding/decoding iterations.

Minor:

  • The abbreviations SOMI and MOMI are used at line 76 before getting defined.
Questions for authors
  1. How is the taxonomy of the dataset defined (i.e. what objects to include)?

  2. Why are two different image generation models used to generate SOMI and MOMI images as per line 104?

  3. Given that some images have a large number of objects, how many iterations on average needed for the image generation model to produce a correct image?

  4. How would any of the image editing models perform if equipped with the validation in section 2.2 to iteratively do the edit until it converges to the correct edit?

Reviewer_g22vOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates instruction-based multi-instance image editing—specifically instructions that explicitly add or remove multiple object instances (up to 30 instances), potentially across multiple categories and coupled with color and spatial constraints. The authors introduce PGAMIE, an automated data generation and verification pipeline that uses Gemini 2.5 Pro for prompt expansion, synthetic image generators (Nanobanana 2 / Imagen 3) for dense multi-object scene synthesis, SAM 3 + Gemini VQA agreement for dual-stage instance verification, and Imagen 3 mask-based inpainting for object removal (inverting removal pairs to obtain artifact-free object addition pairs, and chaining them for compound add+remove edits). Using PGAMIE, the authors construct the MOMI training set (20,000 samples) and three evaluation benchmarks: MOMI Test (300 synthetic samples), SOMI (200 synthetic single-category samples), and SOMI-Real (200 real-image samples from Open Images V7). Benchmarking 10 state-of-the-art open-weight and proprietary editors reveals severe degradation as object counts exceed 5; fine-tuning FLUX.1 Kontext on MOMI Train boosts its MOMI Test exact Edit Accuracy from 2.0% to 16.3%.

Critical strengths
  1. Systematic stress-testing of a clear capability bottleneck in modern image editors: Existing editing benchmarks rarely test explicit numerical count changes beyond 3–5 instances or compound multi-category add/remove instructions. The paper provides a well-structured diagnostic evaluation across 10 modern editors (including proprietary models like Nanobanana Pro and GPT-Image-1.5), clearly exposing how and why current models fail (e.g., count errors vs. opposite-action vs. unrequested edits) as scene density and requested count scale up.
  2. Clever verification-and-inversion data synthesis pipeline with demonstrated training utility: Requiring exact count agreement between SAM 3 concept segmentation and Gemini 2.5 Pro VQA filters out counting hallucinations in synthetic base images, and reversing verified removal inpainting pairs to supervise object addition avoids background alteration. Fine-tuning FLUX.1 Kontext on the resulting 20k MOMI Train set yields large, consistent gains across both synthetic (MOMI, SOMI) and real-image (SOMI-Real: 20.5% to 43.0%) benchmarks.
Critical weaknesses
  1. Mismatch between synthetic training/testing complexity and real-world evaluation: All complex editing settings in this work—Multi-Category instructions, Compound (simultaneous add + remove) edits, and Color+Spatial attribute combinations—are evaluated exclusively on synthetically generated base images (MOMI Test). The only real-image benchmark (SOMI-Real) is restricted to single-category "Basic" instructions. Furthermore, because removal in MOMI is synthesized via Imagen 3 mask inpainting (and addition is its exact inverse), models fine-tuned on MOMI Train may partially overfit to synthetic inpainting reverse-statistics and binary strict-count metrics without preserving fine-grained real-world scene lighting, shadows, or complex occlusions.
  2. Limited operational scope and lack of visual/perceptual quality metrics: Despite being framed broadly as multi-instance image editing, the pipeline and benchmarks only cover object addition and removal (excluding multi-instance replacement, attribute modification, state changes, or spatial rearrangement). Moreover, evaluation relies almost entirely on VLM-inferred count/attribute accuracy (SAM 3 + Gemini 3.1 Pro parsing); the paper does not report standard perceptual quality, visual realism, or background preservation metrics (e.g., PSNR/SSIM on unedited regions or human visual quality ratings) to verify that the fine-tuned FLUX Kontext model does not degrade visual fidelity while optimizing for count accuracy.
Questions for authors
  1. Visual quality and background preservation of the fine-tuned model (Clarification): When FLUX Kontext is fine-tuned on MOMI Train, does it maintain the visual realism, lighting consistency, and non-target background preservation of the base model? Can the authors provide quantitative background preservation metrics (e.g., masked PSNR/LPIPS) or a human preference study comparing visual quality before and after MOMI fine-tuning?
  2. Real-world multi-category and compound editing (Clarification): Why was SOMI-Real restricted only to single-category Basic instructions? Since Open Images V7 contains multi-category annotations and bounding boxes/masks, could the authors evaluate whether the MOMI-tuned model generalizes to multi-category and compound add+remove instructions on real images?
  3. Evaluator bias toward the data-generation stack (Clarification): Both data construction/verification and final benchmark evaluation rely heavily on the same model family (SAM 3 + Gemini 2.5/3.1 Pro). While Appendix .4 validates the evaluator on a subset with MTurk, how do the authors ensure that Gemini 3.1 Pro does not exhibit systematic leniency or bias when evaluating outputs from Gemini-family image editors (Nanobanana / Nanobanana Pro / Nanobanana 2)?
Reviewer_RMP3OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PGAMIE generates scenes with a target number of objects and keeps a scene only when SAM3 and Gemini VQA agree on the count. Following this, it builds removal edits with SAM3 masks and Imagen3 inpainting, checked by same dual verification. Additions are made by reversing the removals, and compound edits are made by chaining removals. A Kontext LoRA is trained on MOMI-Train achiveing 16.3 % edit accuracy on MOMI-test vs. 14.3 % for Nano Pro.

Critical strengths
  • The paper targets a real, under measured problem of exxact counts at high N, multiple categories, and compound add/remove edits.
  • Smartdesign choices are made. For example, removal is easier than addition, hence additions are obtained through reversal. The remove-all shortcut finding and its mitigation (Appendix 2) are useful.
  • The evaluator is validated against humans, and SAM3 vs Yolo-world comparison uses a clustered bootstrap
  • 10 different baselines are used including comercial models
Critical weaknesses
  • The intro calls the model robust as scene density increases, however at >33 objects it scores 2.8% vs Nano Pro at 8.3% (Sec 4.4)
  • It is also claimed that the model is strongest on removals, but on SOMI removals it scores 15% vs 23-26% for Nano Pro and Nano 2 (Fig 31).
  • The advantage seems to be in-distribution only. MOMI-test shares the generator, inpainter, and 32 of its 42 categories with the training. On SOMI and SOMI-Real, the model scores about half ofNano2 and Nano pro.
  • The pipeline is the claimed contribution, but nothing shows what each stage contributes. Useful comparisons could be i) unverified vs sam only vs dual verification ii) with and without reversal for compositional edits iii) same compute fine tune on generic editing data
  • Only Kontext is finetuned. A second backbone could show whether the data transfers, and provides advantage for stronger zero shot models such as flux klein.
Questions for authors
  • How is background change, color shift penalized ?
  • Fig 42 looks different format than others, is it using different data ? Why is it different ?
  • Why does evaluation only count objects with SAM and not Gemini ? (this is what i understand from lines 260-269, please correct if I am wrong. And make the description clearer)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.