- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces PaintBench, a deterministic benchmark for precise visual editing. It procedurally generates tasks with unique ground-truth outputs and uses pixel-level evaluation across 20 fundamental editing operations. Experiments on 11 image-editing models show generally low performance under the proposed metric, with the best model achieving 17.1% mIoU. The paper further introduces TinyGrafixBench for data visualization editing, whose strong correlation with PaintBench provides additional validation of the proposed framework.
Critical strengths
- Uses unique ground-truth outputs, avoiding the bias and instability of human or VLM-based judges.
- Procedurally generates fresh tasks from random seeds, making the benchmark scalable and resistant to contamination or memorization
- Covers 20 fundamental editing operations with controlled scene variations, enabling fine-grained diagnosis of task difficulty and model-specific weaknesses.
- Demonstrates strong external validity through the high correlation between PaintBench and TinyGrafixBench performance.
Critical weaknesses
- I am somewhat concerned about the generalizability of the benchmark, as PaintBench is largely restricted to procedurally generated geometric scenes and leaves out many challenges encountered in real-world editing, such as texture, lighting, occlusion, semantic boundaries, and natural object deformation. While TinyGrafixBench provides additional validation, it follows a similar procedural construction and deterministic evaluation setup, so the strong correlation between the two may still be insufficient to demonstrate generalization to natural images or realistic editing scenarios.
- The pixel-level mIoU metric may be overly strict for generative editors, which often reconstruct parts of the image and inevitably introduce small, perceptually negligible pixel differences. Consequently, the reported 17.1% best performance may partly reflect limited pixel-level reproduction fidelity rather than an inability to execute the intended edit, potentially overstating the difficulty of precise visual editing.
- The evaluation is limited to native pixel-space editing models and excludes increasingly capable agentic or tool-augmented approaches. Given recent progress in systems such as CanvasAgent [1] and UniWorld-Design [2], evaluating representative agentic editors would provide a more complete picture of current precise visual editing capabilities and better position PaintBench within emerging editing paradigms.
[1] CanvasAgent: Enabling Complex Image Creation and Editing via Visual Tool Orchestration [2] UniWorld-Design: From Pixel Generation to Layer-Native Design
Questions for authors
Please refer to the weaknesses.
5 comments
Thank you to the authors for the detailed rebuttal and additional experiments. However, I remain unconvinced about the practical value of PaintBench. The strong ranking correlations with existing benchmarks do not necessarily establish its validity or demonstrate its unique contribution; if PaintBench produces largely the same model rankings, it remains unclear what additional insights or diagnostic value it offers. Moreover, given that a coding agent already achieves 94.7 mIoU on these procedurally defined tasks, the practical necessity of evaluating native image editors on such tasks remains questionable.
Thanks for the follow-up. Rank correlation alone does not necessarily establish validity or a unique contribution. We included it only to answer your earlier question about transfer beyond procedural scenes.
PaintBench gives an exact, deterministic check on a kind of edit that image models are already marketed and used for. Google presents Nano Banana for "precise local edits," infographics and diagrams, and product mockups built "from a single design template" [1, 2]. In that kind of graphic-design work, an edit has a right answer. A recolored element has to match the requested color, a moved shape has to land in the right spot, and the rest of the image has to stay untouched. PaintBench measures whether models actually do that.
Existing benchmarks miss these errors because their judges miss them. ImgEdit and GEdit-Bench both use GPT-4.1 as their judge. We just ran a new test of it on 300 PaintBench problems, scoring the correct answer alongside answers rendered by our generator with a single clear mistake, such as rotating the wrong way, scaling by the wrong factor, or editing the wrong shape. GPT-4.1 gave full instruction-compliance marks to 42% of these wrong answers and scored the correct answer higher in only 48% of pairs (6% for wrong-direction rotations). On real outputs from Nano Banana 2.1, a yes/no GPT-4.1 judge accepted 95% of edits with zero mIoU, the same rate as for its best edits. A stronger reasoning judge (GPT-5.6 Luna) narrows the gap but still accepts 27–43% of zero-mIoU outputs and often misses unrequested removals. PaintBench catches every one of these cases directly.
Why we test image editors directly instead of coding agents. Image editors are the tools being offered for these tasks, so we want to know how reliable they are on exact instructions. We ran a coding agent as a check that each problem can be solved from the image and instruction alone, and its failures helped us find and fix two ground-truth bugs. Writing code works on PaintBench because our scenes have clean, flat shapes. Scripting an exact edit on a photo, like changing a product to a specific brand color while keeping its shading, is far harder, so native editors still need this precision.
Findings that require exact ground truth. Judge scores may struggle to pick these up, and these factors are hard to control in a standard editing benchmark, where scenes are photographs and an edit has many acceptable results.
- Holding the target fixed, adding distractor shapes does not lower edit accuracy for three of four models, while mIoU drops as more shapes have to be preserved.
- Over four chained edits, GPT Image 2.5 Sunburst drops from 39.7 to 13.6 mIoU when editing its own outputs, compared with staying near 40 when each step starts from the correct image.
- Fine-tuning on generator data raises preservation accuracy to 97.3% while edit accuracy reaches only 49.6%.
[1] https://developers.googleblog.com/en/introducing-gemini-2-5-flash-image/
[2] https://deepmind.google/models/gemini-image/flash/
W1: Generalization beyond procedural scenes.
We agree PaintBench leaves out texture, lighting, occlusion, and semantic boundaries. Because those factors rarely admit a single pixel-exact answer, deterministic scoring cannot cover them by design, and the revision states this scope in the introduction and limitations. PaintBench complements natural-image benchmarks by isolating whether a model can execute a fully specified edit accurately without disturbing unedited regions, which judge-based evaluations cannot verify. To test whether this precision transfers to natural photos, we ran all 15 models on two natural-image editing benchmarks for this rebuttal, extending our TinyGrafixBench validation ($R^2 = 0.91$ in the submission and 0.94 with the newer models). PaintBench rankings correlate strongly with the full ImgEdit basic suite (737 edits, GPT-4.1 judge, official scripts, Spearman $\rho = 0.81$, $p < 0.001$) and with GEdit-Bench (606 English requests, $\rho = 0.83$, $p < 0.001$). Because API providers withhold outputs on some natural-image prompts due to safety filters, recitation checks, or provider errors, we score all models on the items returned by every model (655 of 737 ImgEdit items and 499 of 606 GEdit items) so provider policy does not confound editing ability. PaintBench thus measures a specific precision capability that partly transfers to natural-image editing, rather than serving as a general editing score.
W2: Is pixel-level mIoU overly strict for generative editors?
The revision separates edit execution from background reconstruction through three analyses.
Strict preservation is deliberate for settings such as design files, charts, and documents, where unrequested changes elsewhere in the image are real errors. Reporting edit and preservation accuracy next to mIoU (Appendix F) lets each be read independently, and even ignoring preservation entirely, models get roughly half of the edit region wrong.
W3: Agentic / tool-augmented editors (CanvasAgent, UniWorld-Design).
We evaluated a coding agent (Claude Code with Claude Opus 5.5) that, given the input image and instruction, writes and runs image-processing code to return the edited image. Across all 1,920 problems it reaches 94.7 mIoU (edit-region accuracy at $\Delta E_{76} \leq 5$ of 97%, preservation 99.9%), versus 23.9 for the best native editor, and on TinyGrafixBench it reaches 80.0 versus 23.3. The agent sees only the input image and instruction, and an audit of all its transcripts found no access to answer files. The tasks are therefore solvable through code, pinpointing native pixel-space execution as the bottleneck PaintBench measures. We include these results as a separate reference in the revision. We could not evaluate the two cited systems directly because neither has released code or model weights, and UniWorld-Design performs layer decomposition rather than instruction-based editing. We would be glad to evaluate both if runnable releases become available.
Thank you for the constructive feedback, and we hope these new results address your concerns.