acceptodds
Under review as a conference paper at ICLR 2027

PaintBench: Deterministic Evaluation of Precise Visual Editing

Abstract

While current multimodal models are proficient at open-ended visual editing, executing precise single-answer edits remains an important obstacle. To probe this challenge, we introduce PaintBench, a dynamically scalable benchmark targeting 20 fundamental precise visual editing operations across four categories: geometric transformation, structural manipulation, color change, and symbolic reasoning. Procedural generation with configurable complexity enables an effectively infinite, contamination-resistant evaluation suite, and deterministic pixel-level evaluation eliminates reliance on bias-prone judge models. Across 11 image editing models, we find overall low performance, with the current highest-performing industry leader scoring only 17.1% (mIoU). Task decomposition reveals especially challenging operation types (geometric transformation, most structural manipulation, formula-based color change) and model-specific specializations. Fine-grained benchmark diagnostics further show performance degradations induced by scene variations in object count, background complexity, color scheme, and edit-region size. To test generalization of PaintBench scores to applied task performance, we create a procedural, deterministic evaluation for data visualization editing (TinyGrafixBench) and find strong linear correlation with PaintBench scores (R-squared = 0.91, p < 0.001). Altogether, PaintBench provides a rigorous foundation for measuring and driving progress in precise multimodal visual editing.

Primary area:
datasets and benchmarks
Keywords:
benchmark, image editing, precise visual editing, deterministic evaluation, procedural generation, multimodal generative models, pixel-level evaluation, unified multimodal models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DmjyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PaintBench, a deterministic benchmark for precise visual editing. It procedurally generates tasks with unique ground-truth outputs and uses pixel-level evaluation across 20 fundamental editing operations. Experiments on 11 image-editing models show generally low performance under the proposed metric, with the best model achieving 17.1% mIoU. The paper further introduces TinyGrafixBench for data visualization editing, whose strong correlation with PaintBench provides additional validation of the proposed framework.

Critical strengths
  1. Uses unique ground-truth outputs, avoiding the bias and instability of human or VLM-based judges.
  2. Procedurally generates fresh tasks from random seeds, making the benchmark scalable and resistant to contamination or memorization
  3. Covers 20 fundamental editing operations with controlled scene variations, enabling fine-grained diagnosis of task difficulty and model-specific weaknesses.
  4. Demonstrates strong external validity through the high correlation between PaintBench and TinyGrafixBench performance.
Critical weaknesses
  1. I am somewhat concerned about the generalizability of the benchmark, as PaintBench is largely restricted to procedurally generated geometric scenes and leaves out many challenges encountered in real-world editing, such as texture, lighting, occlusion, semantic boundaries, and natural object deformation. While TinyGrafixBench provides additional validation, it follows a similar procedural construction and deterministic evaluation setup, so the strong correlation between the two may still be insufficient to demonstrate generalization to natural images or realistic editing scenarios.
  2. The pixel-level mIoU metric may be overly strict for generative editors, which often reconstruct parts of the image and inevitably introduce small, perceptually negligible pixel differences. Consequently, the reported 17.1% best performance may partly reflect limited pixel-level reproduction fidelity rather than an inability to execute the intended edit, potentially overstating the difficulty of precise visual editing.
  3. The evaluation is limited to native pixel-space editing models and excludes increasingly capable agentic or tool-augmented approaches. Given recent progress in systems such as CanvasAgent [1] and UniWorld-Design [2], evaluating representative agentic editors would provide a more complete picture of current precise visual editing capabilities and better position PaintBench within emerging editing paradigms.

[1] CanvasAgent: Enabling Complex Image Creation and Editing via Visual Tool Orchestration [2] UniWorld-Design: From Pixel Generation to Layer-Native Design

Questions for authors

Please refer to the weaknesses.

5 comments

W1: Generalization beyond procedural scenes.

We agree PaintBench leaves out texture, lighting, occlusion, and semantic boundaries. Because those factors rarely admit a single pixel-exact answer, deterministic scoring cannot cover them by design, and the revision states this scope in the introduction and limitations. PaintBench complements natural-image benchmarks by isolating whether a model can execute a fully specified edit accurately without disturbing unedited regions, which judge-based evaluations cannot verify. To test whether this precision transfers to natural photos, we ran all 15 models on two natural-image editing benchmarks for this rebuttal, extending our TinyGrafixBench validation ($R^2 = 0.91$ in the submission and 0.94 with the newer models). PaintBench rankings correlate strongly with the full ImgEdit basic suite (737 edits, GPT-4.1 judge, official scripts, Spearman $\rho = 0.81$, $p < 0.001$) and with GEdit-Bench (606 English requests, $\rho = 0.83$, $p < 0.001$). Because API providers withhold outputs on some natural-image prompts due to safety filters, recitation checks, or provider errors, we score all models on the items returned by every model (655 of 737 ImgEdit items and 499 of 606 GEdit items) so provider policy does not confound editing ability. PaintBench thus measures a specific precision capability that partly transfers to natural-image editing, rather than serving as a general editing score.

W2: Is pixel-level mIoU overly strict for generative editors?

The revision separates edit execution from background reconstruction through three analyses.

  • Edit execution alone. Scoring only the edit region at a lenient $\Delta E_{76} \leq 5$ yields 53% accuracy for Nano Banana 2 and 46% for GPT-Image-2 (and 63% and 57% for the newer Nano Banana 2.1 and GPT Image 2.5 Sunburst). Even at $\Delta E_{76} \leq 10$, where remaining errors are clearly visible color shifts, those two submitted leaders reach only 58% and 55%.
  • Spatial tolerance. Allowing 1, 2, or 4 pixels of bidirectional slack leaves the ranking of all 15 models unchanged apart from one tied pair (Spearman $\rho \geq 0.996$), with a maximum gain of 1.1 points.
  • Reconstruction fidelity. Asked only to copy the input, submitted API models keep 91–94% of pixels within $\Delta E_{76} \leq 5$ and 9–21% exact. Leaving unedited pixels untouched is part of the task, and because the preservation region is much larger than the edit region, background drift lowers mIoU substantially. Pairing exact ground truth in the edit region with each model's actual preservation in the editing tasks yields ceilings of about 30.5, 31.5, and 32.7 mIoU for Nano Banana 2, GPT-Image-2, and Nano Banana 1.

Strict preservation is deliberate for settings such as design files, charts, and documents, where unrequested changes elsewhere in the image are real errors. Reporting edit and preservation accuracy next to mIoU (Appendix F) lets each be read independently, and even ignoring preservation entirely, models get roughly half of the edit region wrong.

W3: Agentic / tool-augmented editors (CanvasAgent, UniWorld-Design).

We evaluated a coding agent (Claude Code with Claude Opus 5.5) that, given the input image and instruction, writes and runs image-processing code to return the edited image. Across all 1,920 problems it reaches 94.7 mIoU (edit-region accuracy at $\Delta E_{76} \leq 5$ of 97%, preservation 99.9%), versus 23.9 for the best native editor, and on TinyGrafixBench it reaches 80.0 versus 23.3. The agent sees only the input image and instruction, and an audit of all its transcripts found no access to answer files. The tasks are therefore solvable through code, pinpointing native pixel-space execution as the bottleneck PaintBench measures. We include these results as a separate reference in the revision. We could not evaluate the two cited systems directly because neither has released code or model weights, and UniWorld-Design performs layer decomposition rather than instruction-based editing. We would be glad to evaluate both if runnable releases become available.

Thank you for the constructive feedback, and we hope these new results address your concerns.

Thank you to the authors for the detailed rebuttal and additional experiments. However, I remain unconvinced about the practical value of PaintBench. The strong ranking correlations with existing benchmarks do not necessarily establish its validity or demonstrate its unique contribution; if PaintBench produces largely the same model rankings, it remains unclear what additional insights or diagnostic value it offers. Moreover, given that a coding agent already achieves 94.7 mIoU on these procedurally defined tasks, the practical necessity of evaluating native image editors on such tasks remains questionable.

Thanks for the follow-up. Rank correlation alone does not necessarily establish validity or a unique contribution. We included it only to answer your earlier question about transfer beyond procedural scenes.

PaintBench gives an exact, deterministic check on a kind of edit that image models are already marketed and used for. Google presents Nano Banana for "precise local edits," infographics and diagrams, and product mockups built "from a single design template" [1, 2]. In that kind of graphic-design work, an edit has a right answer. A recolored element has to match the requested color, a moved shape has to land in the right spot, and the rest of the image has to stay untouched. PaintBench measures whether models actually do that.

Existing benchmarks miss these errors because their judges miss them. ImgEdit and GEdit-Bench both use GPT-4.1 as their judge. We just ran a new test of it on 300 PaintBench problems, scoring the correct answer alongside answers rendered by our generator with a single clear mistake, such as rotating the wrong way, scaling by the wrong factor, or editing the wrong shape. GPT-4.1 gave full instruction-compliance marks to 42% of these wrong answers and scored the correct answer higher in only 48% of pairs (6% for wrong-direction rotations). On real outputs from Nano Banana 2.1, a yes/no GPT-4.1 judge accepted 95% of edits with zero mIoU, the same rate as for its best edits. A stronger reasoning judge (GPT-5.6 Luna) narrows the gap but still accepts 27–43% of zero-mIoU outputs and often misses unrequested removals. PaintBench catches every one of these cases directly.

Why we test image editors directly instead of coding agents. Image editors are the tools being offered for these tasks, so we want to know how reliable they are on exact instructions. We ran a coding agent as a check that each problem can be solved from the image and instruction alone, and its failures helped us find and fix two ground-truth bugs. Writing code works on PaintBench because our scenes have clean, flat shapes. Scripting an exact edit on a photo, like changing a product to a specific brand color while keeping its shading, is far harder, so native editors still need this precision.

Findings that require exact ground truth. Judge scores may struggle to pick these up, and these factors are hard to control in a standard editing benchmark, where scenes are photographs and an edit has many acceptable results.

  • Holding the target fixed, adding distractor shapes does not lower edit accuracy for three of four models, while mIoU drops as more shapes have to be preserved.
  • Over four chained edits, GPT Image 2.5 Sunburst drops from 39.7 to 13.6 mIoU when editing its own outputs, compared with staying near 40 when each step starts from the correct image.
  • Fine-tuning on generator data raises preservation accuracy to 97.3% while edit accuracy reaches only 49.6%.

[1] https://developers.googleblog.com/en/introducing-gemini-2-5-flash-image/
[2] https://deepmind.google/models/gemini-image/flash/

Reviewer_hZNNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  1. Introduces 1,920 procedural editing problems covering twenty operations and eight visual conditions.
  2. Uses deterministic reference images to measure edit correctness and preservation.
  3. Evaluates eleven models and adds 600 chart-editing problems. Results reveal substantial precision failures.
Critical strengths
  1. Reproducible evaluation. Procedural reference images enable precise scoring without a learned judge.
  2. Useful empirical coverage. Multiple operations, models, visual conditions, and uncertainty estimates support detailed failure analysis.
Critical weaknesses
  1. Object count and target size are confounded. Increasing object count also shrinks shapes, so the experiments cannot isolate clutter effects. (Appendix B.5.)
  2. Low scores need calibration. The evaluation does not separate editing errors from reconstruction and output-normalization errors. Strict preservation remains a valid objective.
  3. One baseline configuration is unclear. FLUX.2-Klein is described as distilled, but its reported settings differ from the recommended distilled configuration. (Tables 8–9)
Questions for authors
  1. Does performance decline with object count when target size and edit area remain fixed?
  2. Can you provide input-copy and reconstruction baselines, plus results for outputs matching the target resolution?
  3. Which FLUX.2-Klein checkpoint was used, and do results hold with its recommended settings?
1 comment

W1/Q1: Object count and target size are confounded (App. B.5).

You are right that shape dimensions scale with $1/\sqrt{n}$ (App. B.5). We scaled shapes to keep total covered area roughly constant so the condition changes how many objects a model must track rather than how much of the canvas is covered (at baseline size, 60 non-overlapping shapes would require more than twice the canvas area). Holding edit-region size fixed across five bins in a pooled regression with task, model, and edit-size controls shrinks the highest-count effect from $-3.8$ to $-1.6$ mIoU points, meaning edit size accounts for about 60% of the count association. The remaining drop is concentrated in small edits (7.1 $\rightarrow$ 2.2 mIoU from baseline to highest count at $64^2$–$128^2$ pixels) and absent for large edits. The revision states this confound in §5.4, reports the controlled analysis, reframes the finding as "dense scenes with small targets," and corrects the caption of Table 11, which wrongly stated that object count was the only variable changed.

To isolate the remaining 40%, we ran the fixed-size experiment you proposed. Across nine single-target tasks, we kept the target shape and edit region pixel-identical across conditions while varying only the number of same-sized distractors (3, 10, 25, or 60, with 162 paired problems per count). For three of the four models tested, extra shapes do not make executing the edit harder. From 3 to 60 shapes, edit-region accuracy at $\Delta E_{76} \leq 5$ rises for GPT Image 2.5 Sunburst (61.6% $\rightarrow$ 65.3%) and Nano Banana 2.1 (55.0% $\rightarrow$ 62.3%), changes little for Qwen-Image-Edit (28.1% $\rightarrow$ 25.2%), and declines only for FLUX 3 Klein (30.1% $\rightarrow$ 24.1%). Overall mIoU still drops (Sunburst 28.5 $\rightarrow$ 10.5), yet excluding distractor pixels keeps it flat (28.5 $\rightarrow$ 28.0). The remaining count effect is therefore a preservation cost from reproducing more unedited shapes exactly. Both analyses now appear in Appendix G, referenced from §5.4.

W2/Q2: Calibration via input-copy, reconstruction, and native-resolution baselines.

Thank you for suggesting these baselines. We ran all three for this rebuttal and report them in Appendix F of the revision.

  • Input copy. Returning the input unchanged scores 0.6 mIoU: preservation accuracy is 100%, but edit-region accuracy is only 0.6% ($\Delta E_{76} \leq 5$), so a model cannot score well by copying the input.
  • Reconstruction. On our 96-problem "generate the exact same image" diagnostic, the submitted API models match the input exactly on 9–21% of pixels and within $\Delta E_{76} \leq 5$ on 91–94%. Preservation errors cap mIoU well below 100 even under a perfect edit. Substituting exact ground-truth pixels into the edit region while keeping each model's actual preservation from the editing tasks gives upper bounds of about 30.5, 31.5, and 32.7 mIoU for Nano Banana 2, GPT-Image-2, and Nano Banana 1, versus actual scores of 17.1, 16.4, and 11.2. The revision reports this reference for every model alongside edit and preservation accuracy.
  • Native resolution. Resizing outputs to the target resolution costs at most about 0.1 mIoU overall. Because resized outputs come almost entirely from the two non-square conditions, filtering to a native-only subset would alter the condition mix. Instead, we passed always-native models' outputs through the same resize round trip and rescored them, lowering the affected subset by 0.05–0.4 points.
  • Separating edit from reconstruction errors. Edit-region and preservation-region accuracy were already reported separately for every model and task in Appendix Tables 16–17, 19–20, and 22–23, and the revision adds a per-model summary next to mIoU in Appendix F (Table 24).

W3/Q3: FLUX.2-Klein checkpoint and settings.

Thank you for catching this mismatch. We evaluated black-forest-labs/FLUX.2-klein-9B (the step-distilled checkpoint) with pipeline defaults (50 steps, guidance 4.0) instead of the model card's recommended 4 steps and guidance 1.0. Re-running at 4 steps and guidance 1.0 raises its score from 3.5 to 3.9 mIoU, tying FLUX.1-Kontext and remaining below FLUX.2-dev (4.6), while its TinyGrafixBench score barely changes (3.3 vs. 3.4). Conclusions are unchanged, and we updated Tables 1, 8, and 9 and all derived figures.

Thank you for the sharp diagnostic questions, and we hope these results resolve all three concerns.

Reviewer_3fQPOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PaintBench, a set of image editing tasks generated by code with precise single-answer, covering geometric transformation, structural manipulation, color change, and symbolic reasoning. Leveraging the ground-truth information, the paper measures pixel-level differences to evaluate editing correctness and precise preservation. It shows that even sota image generators like gpt-image-2 still struggle with small-region editing and tasks with complex backgrounds or precise geometric manipulation.

Critical strengths
  • The proposed evaluation framework is well-motivated. Using programming language to create synthetic editing tasks guarantees that there are uniquely defined answers and enables the use of deterministic pixel-level scoring to evaluate both edit correctness and preservation of unchanged regions. This addresses a meaningful gap in evaluations based on subjective judgements or learned reward models.

  • The benchmark spans 20 operations and systematically varies scene conditions, revealing task-specific weaknesses and substantial over-editing that aggregate rankings would obscure. The strong correlation with TinyGrafixBench scores provides additional evidence that these diagnostics capture capabilities relevant beyond synthetic shape editing.

Critical weaknesses
  • A substantial gap remains between the benchmark and practical editing workflows. PaintBench focuses on synthetic geometric operations, while TinyGrafixBench remains limited to procedurally rendered charts. These tasks differ considerably from the practical editing scenarios highlighted by recent models such as GPT Image 2.5, Ideogram 4.5, FLUX 3, and Krea 3. Evaluating a broader set of recent models on realistic tasks would better establish the benchmark’s practical relevance. Furthermore, precise local editing is essential to iterative workflows, yet the paper provides no multi-turn evaluation of accumulated errors, content preservation, or consistency.

  • The paper reviews failures without sufficiently investigating any potential improvement path. Although a benchmark paper need not introduce a new algorithm, this framework already offers a direct way to investigate solutions. The proposed generator could supply sft data, and its ground-truth answers could provide verifiable rewards for reinforcement learning. It would also be informative to test whether simple tools or an agentic harness can address the observed failures, even as a separate baseline outside the native-model evaluation.

Questions for authors

Please refer to the weakness section. I'm very willing to raise the score if my questions are answered.

1 comment

W1: Gap to practical editing workflows, newer models, and multi-turn evaluation.

PaintBench targets the exact-execution component of practical editing that judge-based benchmarks cannot verify, and the revision makes this scope explicit (please see our response to Reviewer Dmjy W1 for natural-image transfer). We evaluated all recent models we could access, adding GPT Image 2.5 (Sunburst and Flare), Nano Banana 2.1, and the open-weight FLUX 3 Klein (Krea 3, Ideogram 4.5, and the FLUX 3 API were unavailable to us). The newest models improve on the submitted ones but remain far from solving PaintBench.

ModelPaintBench mIoUEdit acc. ($\Delta E_{76} \leq 5$)TinyGrafixBench mIoU
Nano Banana 2.1 (new)23.963%18.4
GPT Image 2.5 Sunburst (new)22.857%23.3
GPT Image 2.5 Flare (new)15.343%15.5
FLUX 3 Klein (new, open)8.728%7.1
Nano Banana 2 (best submitted)17.153%15.9
Qwen-Image-Edit (best submitted open)6.725%3.4

Geometric transformations remain the hardest category for the top two models (13.1 and 15.0 mIoU), showing that PaintBench tracks generational progress without saturating.

To test iterative workflows directly, we added two multi-turn experiments. First, a five-round drift diagnostic repeatedly feeds each model its own previous output with the instruction "Generate the exact same image as the input" and scores every round against the original input. Between rounds 1 and 5, pixels within $\Delta E_{76} \leq 5$ of the original drop from 88% to 34% for FLUX.1-Kontext, 63% to 26% for FLUX.2-Klein, and 51% to 11% for Qwen-Image-Edit. Top API models start nearly lossless yet drift steadily, falling from 98.7% to 58.7% for GPT Image 2.5 Sunburst, 97.5% to 54.1% for Nano Banana 2.1, and 92.8% to 43.3% for GPT-Image-2.

Second, we ran a sequential-edit experiment on 48 scenes, applying four distinct edits (recolor, remove, translate, border) to different shapes with exact ground truth after every turn. When each turn starts from the ground-truth previous image, accuracy holds flat across turns at about 40 mIoU for GPT Image 2.5 Sunburst and 30 for Nano Banana 2.1. When each model starts from its own previous output, scores fall every turn, dropping between turns 1 and 4 from 39.7 to 13.6 (Sunburst), 31.6 to 9.7 (Nano Banana 2.1), 15.4 to 0.4 (Qwen-Image-Edit), and 12.5 to 1.1 (FLUX 3 Klein). Small color shifts in earlier edits and unedited regions compound across turns, showing why multi-step workflows depend on the per-step preservation PaintBench measures.

W2: No improvement path (SFT data, verifiable rewards, tool / agent baseline).

As you suggested, we tested fine-tuning on generator data. We LoRA fine-tuned Qwen-Image-Edit-2511 (rank 32, two epochs) on 15,360 PaintBench problems generated from a disjoint seed namespace with no test-set overlap. PaintBench mIoU rises from 6.7 to 22.2, beating every model in our submitted Table 1 (best 17.1) with gains in all four categories. Edit-region accuracy at $\Delta E_{76} \leq 5$ roughly doubles (25.3% $\rightarrow$ 49.6%) and preservation accuracy climbs from 89.2% to 97.3%. Even after training on 15,360 problems from the generator, the model gets only half of the edit region right, pointing to real model limitations rather than unfamiliarity with the task format.

The deterministic evaluator also scores outputs quickly without a judge and reports edit and preservation accuracy separately. Paired with fresh generator problems on every rollout, it provides a direct verifiable reward for RL, which we leave to future work.

On tool use, a coding agent (Claude Code with Claude Opus 5.5) that edits by writing image-processing code reaches 94.7 mIoU on all 1,920 problems and 80.0 on TinyGrafixBench, compared with 23.9 for the best native editor. This confirms that native pixel-space execution is the bottleneck, and we report the agent as a separate reference in Appendix I.

Thank you for suggesting these experiments, which substantially strengthened the paper.

Comments on the paper

1 comment

We thank all three reviewers for their careful feedback. The reviews agree that deterministic, procedurally generated ground truth fills a real gap in image-editing evaluation, and they raise shared questions about how much low scores reflect reconstruction fidelity versus editing failure, how PaintBench relates to natural-image and agentic editing, and whether the framework points toward model improvement. For this rebuttal, we ran nine new analyses and experiments, all included in the revised PDF.

  1. Score calibration (hZNN, Dmjy). We added a 96-problem "generate the exact same image" reconstruction diagnostic, an input-copy baseline, a resize round-trip check, and separate edit-region and preservation accuracies next to mIoU (Appendix F).
  2. Spatial tolerance on the full release (Dmjy, hZNN). Allowing 1, 2, or 4 pixels of bidirectional tolerance leaves the ranking of all 15 models unchanged apart from one tied pair (Spearman $\rho \geq 0.996$), and no model gains more than 1.1 points.
  3. Tool-using baseline (3fQP, Dmjy). A coding agent (Claude Code with Claude Opus 5.5) that edits by writing image-processing code reaches 94.7 mIoU, versus 23.9 for the strongest native editor. Because the tasks are solvable through code, the bottleneck is executing edits directly in pixel space. Native editors remain our focus, and we report the agent separately as a reference.
  4. Natural-image transfer (Dmjy). Across all 15 models, PaintBench rankings correlate strongly with ImgEdit (Spearman $\rho = 0.81$) and GEdit-Bench ($\rho = 0.83$), both with $p < 0.001$.
  5. Object count controlled for edit size (hZNN). Edit size accounts for about 60% of the object-count drop, and a new fixed-size experiment traces the rest to reproducing more unedited shapes exactly.
  6. SFT on generator data (3fQP). LoRA fine-tuning Qwen-Image-Edit-2511 on 15,360 freshly seeded problems disjoint from the test set raises PaintBench mIoU from 6.7 to 22.2, surpassing every submitted model while remaining far from solving the benchmark.
  7. Newer models (3fQP). Nano Banana 2.1 (23.9 mIoU) and GPT Image 2.5 Sunburst (22.8) improve on the best submitted model (17.1) while leaving plenty of headroom. FLUX 3 Klein (8.7) is now the strongest off-the-shelf open model, ahead of Qwen-Image-Edit (6.7).
  8. Multi-turn drift (3fQP). Asked to reproduce their own output five times in a row with no edit requested, the strongest API models keep only 43–59% of pixels within $\Delta E_{76} \leq 5$ of the original (down from 93–99% after one round), and open models keep 11–34%. In a new sequential-edit experiment with four different edits per scene, models given their own previous outputs lose accuracy every turn, whereas models given the ground-truth previous image hold steady.
  9. FLUX.2-Klein at its recommended distilled settings (hZNN). Evaluating with its recommended distilled configuration raises its score from 3.5 to 3.9 mIoU, leaving all conclusions unchanged.

The revised paper also clarifies our scope. PaintBench is a controlled diagnostic of exact edit execution and pixel preservation in native pixel-space editors, complementing benchmarks for open-ended natural-image editing and tool-augmented systems. Exact execution matters wherever edits cannot tolerate drift, including design files, charts, documents, and interfaces. Unified models are increasingly expected to handle these edits directly in pixel space, where judge-based benchmarks cannot verify exactness.

Ground-truth correction. While inspecting the coding agent's errors, we caught two bugs in our geometric-task answers. Rotation answers turned opposite to the instructed direction, and 1-D scaling of tilted shapes ran along the shape's local axes instead of the image axes. We fixed both (70 rotation and 34 scaling answers, keeping inputs and instructions unchanged), rescored every model, and released the corrected answers. Native-model scores shift by at most 0.03 mIoU with no ranking changes. All numbers below use the corrected answers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.