acceptodds
Withdrawn from ICLR 2027

Beyond Plausibility: Verifiable Fine-Grained Image Editing on Structured Assets

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Fine-grained image editing requires more than producing a visually plausible result: an editor must execute the requested attribute change precisely while leaving everything else intact. However, existing benchmarks leave a critical gap between realism and verifiability: benchmarks built on realistic images typically rely on human or VLM judgments, while deterministic evaluation has largely focused on synthetic shape canvases, with application-oriented extensions primarily limited to charts. This makes it difficult to determine precisely how much of a requested edit was executed, where unintended changes occurred, and whether small differences between models reflect genuine editing capability or evaluator uncertainty. To bridge this gap, we present VeriEdit-Bench, a benchmark for fine-grained, instruction-faithful image editing across realistic structured assets with deterministic, four-axis evaluation. Its 1,740 cases are compiled from the source code of 153 SVG graphics, charts, web interfaces, and presentation slides. Controlled source-code edits preserve the original visual context while yielding exact target images, pixel-level edit masks, and explicit edit specifications, enabling reproducible scoring along four axes: edit fidelity, preservation, localization, and magnitude. Evaluating eleven editors, we find that even the strongest model remains far from full credit; rankings for the same recoloring operation reverse between charts and SVG graphics; and outputs with similar pixel-accuracy profiles can still differ substantially in localization and change magnitude. This decomposition yields graded, verifiable feedback and exposes model-specific capability and failure profiles that holistic scores or evaluator-dependent judgments may obscure.

Primary area:
datasets and benchmarks
Keywords:
Image editing, Verifiable benchmark, Structured assets

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_kPwAOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces VeriEdit-Bench, a benchmark for fine-grained image editing on structured visual assets. It contains 1,740 editing cases derived from 153 source assets spanning SVG graphics, charts, web interfaces, and presentation slides. Controlled source-code modifications yield reference targets and edit masks, enabling deterministic evaluation along four axes: fidelity, preservation, localization, and magnitude. The paper evaluates eleven image editors and analyzes operation-specific performance, cross-asset differences, and sensitivity to scoring choices.

Critical strengths
  1. Reproducible benchmark construction. Source-controlled edits provide explicit targets and edit regions, enabling repeatable evaluation of precisely specified changes without relying on human or VLM judgments for the core scoring.

  2. Coverage of multiple structured asset types. Evaluating shared and asset-specific operations across SVG graphics, charts, web interfaces, and slides provides useful insights into performance differences that a single-domain benchmark may overlook.

  3. Extensive supporting analyses. The paper includes controlled perturbations, scoring ablations, parameter-sensitivity analyses, and source-level bootstrap intervals. Reporting individual axes alongside the composite also helps expose different failure patterns.

Critical weaknesses
  1. Pixel-area-based scoring may underweight semantically important errors in small regions. The benchmark targets structured assets such as charts, web interfaces, and slides, where a small change to a number, unit, minus sign, or label can substantially alter meaning. However, Preservation averages pixel agreement across the entire protected region, while Localization filters out small connected changes outside the edit region as rendering noise. Consequently, damage to small but important content may receive only a minor penalty or be omitted from Localization altogether. An output could therefore execute the requested edit while corrupting critical protected content and still receive high scores. The existing controls, which primarily examine color-group modifications and background damage, do not adequately rule out this risk. Controlled tests involving small text, numerical values, and legends, together with object-level or text-level preservation measures, would strengthen the validation.

  2. The composite score exhibits a substantial floor effect, limiting its ability to distinguish partial success and incremental improvements. Appendix Table 11 reports that 52.5%–84.9% of outputs from open-weight models receive a zero composite score. Table 12 further shows that 87% of all zero-composite outputs have nonzero Localization and Magnitude but receive zero Fidelity, which forces the harmonic composite to zero. Fidelity measures agreement at a finite set of color tolerances and subtracts the unchanged-input baseline. An output can therefore move closer to the target without receiving additional credit if that improvement does not increase thresholded agreement; improvements along the other axes likewise remain invisible to the composite when Fidelity is zero. Although the individual axes retain some diagnostic information, the primary ranking metric collapses many potentially different degrees of failure into the same score, weakening the claimed fine-grained feedback. The paper would benefit from examining meaningful progress within zero-score outputs and comparing against continuous target-distance measures, beyond establishing aggregate ranking stability under threshold changes.

Questions for authors

See Weaknesses.

1 comment

Q1: Pixel-area-based scoring may underweight semantically important errors in small regions.

Weakness 1

A1:

This comment identifies a valuable complementary dimension that can be reported separately: whether key text and numerical content retain their meaning. It is related to precise visual execution but asks a different question. It does not require redefining the four axes as measures of semantic correctness. Our axes use explicit visual references to measure target appearance, pixel preservation, edit extent, and change magnitude and direction (Section 3.3; Appendix B.2).

The distinction has a practical basis. A minus sign occupies few pixels but can change the meaning of a value. Assessing that importance requires recognizing the text and its context. P measures how much of the protected region retains pixel agreement. L measures spatial coverage and confinement. Neither weights pixels by the meaning of their text. Visual preservation and semantic correctness can therefore be reported separately: one measures how much of the image is preserved; the other checks whether its information remains correct.

As a supplementary control for this response, we tested your concern by constructing 57 errors in numbers, signs, units, or labels across 17 chart source assets. We first completed the requested edit, then changed one text object that should have remained intact. All constructed outputs still obtained H≥95 and P≥99; 55 had L=100. High visual agreement can therefore coexist with a local semantic error, so semantic correctness requires an additional check. These results define the interpretation of high axis scores. Such scores do not guarantee intact text meaning, but this does not invalidate their measurement of precise attributes and visual changes.

We therefore view the suggestion as a semantic-preservation dimension alongside the four axes, rather than a reason to adjust them to replace semantic judgment. Object- or text-level content checks could supply additional evidence for applications that depend on correct numbers, units, and labels. We thank you for identifying this useful direction. It extends the range of evaluated capabilities while retaining the clear meaning of the visual measurements.

Q2: The composite score exhibits a substantial floor effect, limiting its ability to distinguish partial success and incremental improvements.

Weakness 2

A2:

Using H alone compresses differences in partial success. This is why the paper reports the four-axis profile alongside H. H measures joint performance, while the axes retain the attained level of each capability. If one axis is zero, H is zero, but the other axes can still show coverage, preservation, or change-control ability. Section 3.3, Evaluation Protocol, defines this reporting scheme.

Section 4.4, Figure 5 demonstrates the distinction. Complete images and per-tolerance accuracy curves are in Appendix A.11, Figures 19–20. On the same web recoloring task, FLUX.2-dev changes the requested green elements, but its purple is shifted toward magenta. Step1X-Edit leaves the target elements green and changes unrelated page regions. Neither earns fidelity credit, and both have H=0. Their axis scores nevertheless distinguish the execution patterns:

Real output on the same taskF: fidelityL: localizationM: magnitudeH: composite
FLUX.2-dev0.0082.5697.380.00
Step1X-Edit0.0060.9459.020.00

The former shows better target coverage and color-change control but needs more accurate final colors. The latter first needs to execute the edit in the intended region. Equal H therefore does not make these capability states indistinguishable in our full reporting scheme. The axes provide interpretable feedback beyond the composite.

A supplementary continuous-distance check of the existing outputs for this response also found that some zero-score outputs are closer to the target. This further supports interpreting zero composites together with axis-level diagnostics, rather than as an absence of all progress.

Our reporting separates joint performance from partial success: H summarizes the former, while the axes expose capability differences and improvement directions. The real-output example demonstrates the value of this profile beyond reporting a composite alone.

Finally, we sincerely thank the reviewer’s feedback. We respectfully hope that the reviewer could reevaluate our work given the responses addresing your main concerns.

Reviewer_H8NJOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces VeriEdit Bench, a benchmark for evaluating fine grained, instruction faithful image editing on structured visual assets.

Critical strengths
  • Deterministic evaluation across diverse structured visual assets.

  • The experimental analysis provides useful insights into the limitations of current raster image editing models.

Critical weaknesses
  • This paper provides limited methodological novelty over existing evaluation frameworks. The proposed evaluation dimensions, particularly fidelity, preservation and localization are already well established in prior image editing benchmarks. PaintBench also provides deterministic evaluation using reference images and pixel level comparisons. While the paper extends this setting to different structured visual assets and introduces an additional magnitude based analysis, it is not sufficiently clear what fundamentally new evaluation capability is introduced. The experiments show that the proposed metrics can distinguish certain failure patterns that existing measurements may overlook, but the significance of these additional distinctions is not fully established. This raises concerns about whether the contribution goes substantially beyond combining existing evaluation principles in a new benchmark. Moreover, the evaluation seems incomplete since it does not consider multimodal reasoning models (e.g., GPT 6 Astra or stronger baselines) or agentic editing pipelines that can interpret and modify structured representations. Hence, it is hard to justify the complexity of the benchmark in the case of stronger models.

  • The benchmark is constructed from structured representations such as SVG, charts, HTML/CSS and slides. However, the evaluated models operate exclusively on rasterized images. While the paper explicitly motivates this setting and distinguish it from source based editing, many of the considered operations can be performed directly and precisely on the underlying structured representations. The evaluation does not consider source aware editing systems or multimodal reasoning agents capable of manipulating these representations. As a consequence, it is unclear how representative the reported performance limitations are of modern structured visual editing systems more broadly. The results establish the difficulty of these tasks for the selected raster editing models, but do not establish whether the same limitations persist when more suitable editing representations or workflows are available. Finally, it is worth noting the converting SVG to image and then editing image and converting back to SVG can be very lossy and hence this also raises concerns on the utility of the benchmark.

  • The paper demonstrates that the 4 evaluation dimensions can identify different editing failures, but it remains unclear how these diagnostics can be used to improve image editing models in practice. The analysis primarily highlights differences in metric scores, failure patterns and model rankings without demonstrating whether these insights can guide better model selection, targeted improvements or more effective model development. The practical value of the additional evaluation dimensions should be better established, especially given their overlap with existing metrics. Without such evidence, it is difficult to assess whether the proposed framework provides actionable insights or primarily offers a more detailed description of existing model limitations.

Questions for authors

My questions (predominantly related to the above weaknesses):

  • PaintBench already supports deterministic pixel level evaluation, while prior benchmarks consider fidelity, preservation and localization. Could the authors clarify which capabilities of the proposed framework cannot be captured by existing evaluation methods? Particularly what additional information does the magnitude dimension provide beyond the existing measurements and why is this information important for evaluating editing quality?

  • Is there any empirical evidence that the proposed diagnostics provide actionable value for improving editing systems?

  • How do the authors justify evaluating structured visual assets exclusively through raster editing? In my opinion, this affects the benchmark's practical relevance.

  • Could the authors clarify whether the benchmark is intended specifically for raster image editing or for structured visual editing more broadly? If the latter, what evidence supports the generality of the conclusions beyond the selected model family? A comparison with stronger source aware or tool assisted editing systems would help establish the practical difficulty and remaining headroom of the benchmark.

  • Have the authors evaluated how well the scores correlate with human assessments of instruction satisfaction and preservation on actual model outputs? Such an analysis would help determine whether low fidelity scores primarily reflect meaningful editing failures or in some cases visually acceptable deviations from the exact reference.

  • How sensitive are the reported model comparisons to differences in inference resolution, preprocessing and output resizing?

8 comments

Q1: How do the authors justify evaluating structured visual assets exclusively through raster editing? In my opinion, this affects the benchmark's practical relevance.

Original Question 3; Weakness 2 (image input and source editing).

A1:

We respectfully disagree that the ability to perform an operation directly in source code undermines the value of evaluating image editors. Our goal is to measure the precise editing ability of foundation image-editing models. Structured assets provide exact, reproducible target images while retaining visual context from practical applications.

Section 1 states this motivation; Section 3.1 and Figure 2 give the construction process:

Our key insight is that structured visual assets provide editable source representations that explicitly define their visual attributes.

Controlled source edits give exact before-and-after renders and change regions. They let us test whether a model executes the specified operation. PaintBench also provides exact references, primarily on shape canvases with an extension to charts. Our constructed SVG graphics, charts, web interfaces, and slides retain typography, layout, and component context. This enables comparison of the same editing abilities across these application types.

Precise source control is what makes a reliable reference possible. The known attribute edit and its rendered outcome allow direct verification of the requested change. The introduction also states:

The source is used only to construct and score tasks; the evaluated editor receives the rendered image and the instruction alone.

This construction benefit agrees with the strength you identify as “Deterministic evaluation across diverse structured visual assets.” Reviewer kPwA likewise notes in Strength 1 that source-controlled edits provide “explicit targets and edit regions, enabling repeatable evaluation of precisely specified changes.”

Direct source editing is therefore the basis for constructing verifiable references, not a reason to dismiss evaluation of image-editing ability. Structured assets give us a way to measure precise execution by foundation image editors against known targets.

Q2: Finally, it is worth noting that converting SVG to image and then editing image and converting back to SVG can be very lossy and hence this also raises concerns on the utility of the benchmark.

Weakness 2 (conversion back to SVG; excerpt).

A2:

We respectfully clarify a factual misunderstanding: the benchmark has no step that converts the edited output back to SVG. The cited round-trip loss does not occur in our evaluation pipeline.

Section 3 and Figure 2 specify that the editor receives an image and instruction and returns an image. The remaining information is used to verify the specified transformation. Controlled source edits produce the input and target renders. The scorer evaluates the returned image directly; it does not convert that output to SVG.

Loss from converting an image back to SVG therefore does not constitute a defect in this pipeline or its utility. A1 explains why structured assets are used. Here, the distinction is between using source code to construct a reference and converting a model's output back into source code.

Q3: Could the authors clarify whether the benchmark is intended specifically for raster image editing or for structured visual editing more broadly? If the latter, what evidence supports the generality of the conclusions beyond the selected model family? A comparison with stronger source aware or tool assisted editing systems would help establish the practical difficulty and remaining headroom of the benchmark.

Original Question 4; Weakness 1 (stronger systems); Weakness 2 (scope).

A3:

We respectfully disagree that the absence of a stronger structure-based or tool-assisted system is sufficient to make this evaluation incomplete. How the assets are constructed and which models the study evaluates are distinct questions. Our goal is to measure precise editing by foundation image editors. Structured assets provide controlled, reproducible references for that measurement.

Section 1 states this design rationale; Section 3.1 and Figure 2 give the construction process:

Our key insight is that structured visual assets provide editable source representations that explicitly define their visual attributes.

Editable source lets us specify the attribute change and render its target. The resulting reference tests whether a general image editor executes the request. The model still receives only the image and instruction.

Your original Question 4 conditions the request for broader generality on “If the latter.” Our study addresses the former: raster image editing. We measure the eleven tested editors on verifiable tasks. The cross-asset reversal under a shared recoloring operation exposes capability differences that a single asset type cannot summarize (Section 4.3, Figure 4). Reviewer kPwA also recognizes this in Strength 2, noting “performance differences that a single-domain benchmark may overlook.”

Stronger systems can be evaluated under the same input conditions, and higher scores would demonstrate progress in precise editing. Recovering structure from an image or using tools is an evaluable approach, but studying that approach is outside the scope of the present model study. Giving a system the task's hidden source code instead changes the available information. Reporting the abilities of current image editors does not require all possible editing workflows to share their limitations.

Our contribution is to use structured assets to obtain exact references, directly measure image-editing ability, and identify specific weaknesses. Showing that every other editing approach is equally difficult is not a prerequisite for this contribution.

Q4: PaintBench already supports deterministic pixel level evaluation, while prior benchmarks consider fidelity, preservation and localization. Could the authors clarify which capabilities of the proposed framework cannot be captured by existing evaluation methods? Particularly what additional information does the magnitude dimension provide beyond the existing measurements and why is this information important for evaluating editing quality?

Original Question 1; Weakness 1 (added measurement information).

A4:

Similar metric names do not imply the same measurement basis or information. We ground the dimensions in the same specified edit and exact reference, then test whether they reveal execution differences missed by existing measurements.

The introduction distinguishes this from evaluations based on human, perceptual-model, or VLM judgments: “Such scores do not necessarily provide direct, deterministic verification against an exact reference.” PaintBench already provides exact targets and pixel-based comparisons. We build on that basis, extend a common evaluation to attribute edits in SVG graphics, charts, web interfaces, and slides, and distinguish target appearance from target coverage and input-relative change.

This reveals two findings: model advantages can reverse across asset types under the same recoloring operation, and similar target accuracy can hide different coverage and change-control failures. They show the added value of cross-asset evaluation and axis-level measurement, respectively. Section 4.3, Figure 4, compares 455 shared recoloring cases: FLUX.2-dev leads FireRed-Edit on charts but trails it on SVG. The reversal remains after balancing edit-region size. This reveals differences that one asset type cannot summarize and identifies asset–operation groups that deserve separate attention in training-data construction.

L checks precise target coverage: did the model change the pixels that the reference edit requires? It also constrains changes outside the allowed region. P checks preservation outside that region and does not test whether the target region was edited. Section 4.1 already reports that unchanged inputs receive P=100 and L=0 across all 1,740 cases. Missing part of the target may preserve high P but reduce L. Editing the right region to the wrong color can give high L; F evaluates the target appearance.

The controls already reported in Appendix A.6, Table 8, on the same 110 cases show why similar F/P scores can require different improvements:

Constructed outputF: fidelityP: preservationL: localizationM: magnitude
Set half the target region to the target values50.17100.0066.6575.03
Cover the full target region, with 1.5 times the reference change49.0799.84100.0093.51

The first needs to complete the missing region; the second needs to calibrate change strength. L identifies coverage, while M provides feedback relative to the reference change.

For recoloring, M measures control over distance and direction in color space; F measures attainment of the final target appearance. Distance means the amount of color change, not movement across the canvas. M compares input-to-output and input-to-target color changes at corresponding pixels. It can distinguish whether the requested amount and direction were followed even when the exact target color was missed. Brightening and darkening by the same amount, for example, have opposite directions.

Section 4.4, Figure 5, provides a real example; Appendix A.11, Figures 19–20, gives the complete images and curves. FLUX.2-dev changes the requested green elements, but its purple is shifted toward magenta. Step1X-Edit leaves the target elements green and changes unrelated page regions. Both have F=0. Their L scores are 82.56 and 60.94, and their M scores are 97.38 and 59.02. The former needs more accurate final colors; the latter first needs to edit the target region. In a supplementary analysis of the existing 607-case panel outputs (Section 4.4; Appendix A.11), additionally matching PaintBench scores and F/P/L still leaves 70 pairs with an M gap of at least 20 points, spanning 28 cases, 25 source assets, and 7 asset–operation groups.

Reviewer kPwA also recognizes axis-level reporting in Strength 3:

Reporting individual axes alongside the composite also helps expose different failure patterns.

The real outputs and known-error controls above provide specific evidence for that value. The contribution is not simply to rename dimensions. It is to distinguish target coverage and change control behind similar target accuracy, using exact references across asset types.

Reviewer_1JqWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

VeriEdit-Bench evaluates precise raster-image editing on SVG graphics, charts, web interfaces, and slides. Editable sources generate reference images and edit masks, while the editor receives only the rendered input and an instruction. The benchmark comprises 1,740 tasks from 153 sources and reports fidelity, preservation, localization, and magnitude profiles, together with a harmonic summary. Experiments include eleven editors, controlled scorer checks, and comparisons of diagnostics on identical outputs.

Critical strengths

Source-based target construction makes the requested changes and protected content directly inspectable. The shared protocol across four structured asset types is useful for reproducible evaluation.

The additional diagnostics are not merely asserted. Among 416 same-case model pairs with closely matched PaintBench profiles, 223 differ by at least 20 points in localization or magnitude (Sec. 4.4). The example in Fig. 5 gives these differences a concrete error interpretation.

Controlled perturbations, independent color-attainment analysis, source-level bootstrap intervals, and aggregation/sensitivity checks provide substantial methodological support. Secs. 4.3 and 4.5 also show cross-asset ranking reversals and that color attainment can coexist with preservation/localization failures.

Critical weaknesses

The incremental measurement contribution remains limited in scope. PaintBench [1] already establishes deterministic rendered targets and edit/preservation accounting, while FigEdit [2] provides specification-based chart editing evaluation. VeriEdit-Bench's additional contribution is therefore the common asset protocol and the extra localization/magnitude interpretation. The controlled checks establish that these axes respond to their intended errors, but the paper's main conclusions still largely characterize familiar execution, preservation, and spatial-control failures within prescribed edits. I recognize the extra information, yet am not convinced that its demonstrated scientific utility constitutes a sufficiently substantial advance over the existing precise-editing evaluation paradigm. This concerns the increment shown in the paper, not an omitted-preprint comparison: the paper already makes the relevant comparison.

The reach of the demonstrated diagnostic increment remains limited. The 416 matched pairs are selected from 33,385 same-case pairs, with the 223 large differences spanning 68 cases and 56 sources (p. 8). This valid existence demonstration does not by itself establish how often the extra axes materially change conclusions across the task distribution. I also acknowledge the broader cross-asset reversals in Sec. 4.3 and the color-attainment/preservation distinction in Sec. 4.5: the evidence is not confined to matched pairs. These are useful descriptive findings, but they remain tied to the constructed source/operation families and do not establish that asset type itself causes the differences. The manuscript appropriately acknowledges that causal limit. Fig. 5 also has zero fidelity for both outputs: the profile distinguishes their failure types, whereas H collapses both to zero. The paper acknowledges this design, so I do not treat it as a hidden defect; rather, the strongest contribution resides in descriptive profiles, not a broadly improved scalar evaluation. On 153 constructed sources and prescribed operations, I regard the overall contribution as a carefully validated but limited extension of diagnostic coverage. The restriction matters in conjunction with the modest protocol innovation, rather than constituting a standalone flaw. I am not asking it to represent all natural-image editing.

References:

Questions for authors
  1. Secs. 4.3 and 4.5 provide useful cross-asset and color-attainment findings. Which conclusions specifically require the added localization/magnitude axes, and which follow from asset coverage or edit/preservation scores? Please distinguish these sources of contribution using the existing analyses.

  2. Can the present matched-pair and source-level analyses quantify the coverage of consequential extra diagnostics across asset/operation groups, while keeping profile conclusions separate from those obtainable from H? A bounded reanalysis of existing outputs would be sufficient; I am not requesting a new broad dataset.

3 comments
AuthorsOpenReview ↗

Response to Original Question 1; Weakness 1 (measurement contribution).

Q1: Secs. 4.3 and 4.5 provide useful cross-asset and color-attainment findings. Which conclusions specifically require the added localization/magnitude axes, and which follow from asset coverage or edit/preservation scores? Please distinguish these sources of contribution using the existing analyses.

A1:

The findings have distinct sources. The cross-asset ranking changes in Section 4.3 come primarily from evaluating a shared operation across assets. The distinction between color attainment and preservation in Section 4.5 comes from the separate color diagnostic and P; L identifies failures in edit extent. The execution differences behind similar pixel-accuracy profiles in Section 4.4 are revealed by L/M.

L primarily asks whether the required locations were edited. M asks how much the output changed from the input and whether the change followed the reference direction. They separate different execution failures even when the target appearance is incorrect.

L and P examine different regions and questions. P measures preservation outside the allowed region. An unchanged input can therefore obtain P=100 without executing the edit. Section 4.1 already reports that unchanged inputs receive P=100 and L=0 across all 1,740 cases. An output that misses part of the target can also retain high P, while L detects incomplete coverage. Conversely, editing the correct region to the wrong color can yield high L. F measures agreement with the target appearance.

The controls already reported in Appendix A.6, Table 8, on the same 110 cases show why similar F/P scores can call for different improvements:

Constructed outputF: fidelityP: preservationL: localizationM: magnitude
Set half the target region to the target values50.17100.0066.6575.03
Cover the full target region, with 1.5 times the reference change49.0799.84100.0093.51

The first output needs to complete the missing region. The second needs to calibrate its change strength. L indicates coverage, while M provides feedback relative to the reference change.

For recoloring, M measures control over distance and direction in color space; F measures attainment of the final target appearance. Distance here means the amount of color change, not an object's displacement on the canvas. M compares the input-to-output and input-to-target color changes at corresponding pixels. It can thus distinguish how closely the changes follow the requested amount and direction even when neither output reaches the exact color. Brightening and darkening by the same amount, for example, have opposite directions.

Section 4.4, Figure 5 provides a real example. Appendix A.11, Figures 19–20 provides the complete images and accuracy curves. FLUX.2-dev edits the requested green elements, but its purple is shifted toward magenta. Step1X-Edit leaves the target elements green and repaints unrelated page regions. Both have F=0. Their L scores are 82.56 and 60.94, and their M scores are 97.38 and 59.02. The former needs more accurate target colors; the latter first needs to edit the intended region. In a supplementary analysis of the existing 607-case panel outputs (Section 4.4; Appendix A.11), additionally matching PaintBench scores and F/P/L still leaves 70 output pairs with an M gap of at least 20 points. They span 28 cases, 25 source assets, and 7 asset–operation groups.

M therefore distinguishes control over the visual change from exact target attainment. L adds coverage and confinement information. Together, they separate similar accuracy scores into specific capability gaps that call for different improvements.

AuthorsOpenReview ↗

Response to Weakness 1 (scientific utility; excerpt).

Q3: I recognize the extra information, yet am not convinced that its demonstrated scientific utility constitutes a sufficiently substantial advance over the existing precise-editing evaluation paradigm.

A3:

The scientific utility is to turn an editing failure into a testable capability target and to use task-specific strengths for model selection. Axis and color diagnostics support the first use. A selection experiment based on asset–operation groups tests the second.

First, color analysis separates control over the change from attainment of the exact target. The values below are taken from Appendix A.10, Table 22: GPT and the four open-weight models with the highest M on the same 621 color cases. Both columns are case means on a 0–100 scale. Color attainment is a separate diagnostic and does not enter M.

ModelM: magnitude and directionColor attainment
GPT-Image-293.4661.47
FLUX.2-dev79.476.44
Qwen-Image-Edit-251175.555.48
FLUX.2-klein75.293.31
FireRed-Edit73.813.04

GPT's M is 13.99 points above the highest open-weight score. Its outputs more closely match the requested amount and direction of color change. Some open-weight models already have relatively high M but very low precise color attainment. Together with the real example in A1, this reveals a gap between coarse color control and exact execution. It identifies a specific target: improving final color accuracy on top of existing color-change capability.

The cross-asset differences in Section 4.3 further suggest where to build targeted training data. Weak asset–operation groups can be paired with explicit color instructions and accurate targets. Axis scores can then test whether target precision improves while surrounding content is preserved. This is a concrete, testable direction derived from the measurements.

Second, task-specific strengths can guide model selection. This experiment selects editors using H within asset–operation groups, so it tests the utility of grouped evaluation. The controls in A1 separately establish the additional information from L/M.

For example, should a new chart recoloring task use FireRed, the strongest open-weight model overall, or FLUX.2-dev, selected from other chart recoloring cases? For this additional analysis, we reuse the model outputs underlying Table 1. We leave out one complete source asset, choose models using the other 152 sources, and evaluate the chosen models on existing outputs from the held-out source. Repeating this for all 153 sources covers all 1,740 cases. The held-out source's scores do not enter its selection. We do not choose the best output after seeing its case score.

The two candidate sets answer separate questions: selection among open-weight models, and selection when GPT is also available. H is averaged equally across the paper's 22 asset–operation groups.

Available modelsOne model for all tasksSelection by asset and operationChange in H and 95% interval
10 open-weight modelsFireRed-Edit, 21.8526.05+4.20 [2.88, 5.48]
All 11 models, including GPTGPT-Image-2, 47.1346.93−0.21 [−0.81, 0.44]

The same rule produces the following choices and held-out results:

Task, with open-weight candidatesModel selected using other sourcesH using FireRed throughoutH after task-specific selection
Chart recoloringFLUX.2-dev5.0517.39
Chart line widthQwen-Image-Edit-251114.2017.50
Slide opacityBAGEL1.7529.61
SVG recoloringFireRed-Edit (7 sources); Qwen-Image-Edit-2511 (28 sources)15.5310.69

Selection does not improve every task, but it improves the overall score by 4.20 points among the 10 open-weight models. Selection by asset type alone gives 20.93; adding operation information raises it to 26.05. The usable complementarity is therefore operation-specific. Intervals resample complete source assets while keeping the obtained selection decisions fixed.

When GPT is available, overall selection chooses it in all 153 held-out runs. Task-specific selection also chooses it for 1,610/1,740 cases and yields no clear additional gain. GPT has broadly strong performance within this candidate set.

These two uses have separate evidence. Axis and color analyses identify testable capability targets. Grouped evaluation makes task-specific strengths usable for selection and, among the ten open-weight candidates, improves performance on sources excluded from each selection decision.

AuthorsOpenReview ↗

Response to Original Question 2; Weakness 2

Q2: Can the present matched-pair and source-level analyses quantify the coverage of consequential extra diagnostics across asset/operation groups, while keeping profile conclusions separate from those obtainable from H? A bounded analysis of existing outputs would be sufficient; I am not requesting a new broad dataset.

Original Question 2; Weakness 2 (coverage and composite score).

A2:

The additional diagnostics extend across all four asset types and multiple sources. They distinguish execution patterns among outputs with similar existing scores, rather than only explaining a few selected examples.

We reuse the 607-case panel already reported in Section 4.4 and Appendix A.11, Table 24. It covers all 153 source assets and 22 asset–operation groups, giving 33,385 same-case model pairs. We require a PaintBench mean intersection-over-union gap of at most 1 point. Both editing and preservation accuracy must differ by at most 2 points at every color tolerance from 0 to 10. For this response, we additionally require F and P to each differ by at most 2 points. This leaves 289 pairs. Of these, 157 (54.3%) still differ by at least 20 points in L or M. They span 49 cases, 40 source assets, and 11 asset–operation groups. The 54.3% figure refers specifically to these 289 matched pairs.

Asset typeMatched pairs with an L or M gap of at least 20 points
SVG graphics117
Charts7
Web interfaces23
Slides10
Total157

The differences occur in all four asset types. SVG palette replacement accounts for 87 pairs, showing that multi-object recoloring is a particularly concentrated setting for these diagnostics. In 115 of the 157 pairs, both outputs have H=0. At least one axis receives no credit, yet the profiles still distinguish edit extent and change magnitude. This follows the reporting scheme in Section 3.3: H summarizes joint performance, while the axes retain the attained level of each capability. The additional information is an execution difference that remains visible when composites are equal. The real example discussed in A1 and Section 4.4 shows how such differences identify specific improvement targets.

The added diagnostics thus have coverage across sources and asset types. Their practical value is to preserve execution differences hidden by equal composites, so different failures can receive different improvement priorities.

Q4: These are useful descriptive findings, but they remain tied to the constructed source/operation families and do not establish that asset type itself causes these differences. The manuscript appropriately acknowledges that causal limit.

Weakness 2 (source coverage and causal interpretation; excerpt).

A4:

The supported conclusion is that performance on one asset set cannot substitute for measurement on another, even when the editing operation is shared. The common protocol makes this comparison possible.

Section 4.3, Figure 4, fixes recoloring and compares the same models on 455 cases across four asset types. FLUX.2-dev leads FireRed on charts but trails it on SVG. Neither source-group interval for these gaps crosses zero. The reversal also remains after equally weighting small and large edit regions (Appendix A.9, Tables 18 and 20). Thus, even the same named operation can expose capability differences that one asset set alone does not capture.

These findings describe the tested source–operation combinations. Asset type, design, layout, and source were not independently manipulated. We therefore report cross-set differences, rather than treating asset type as an isolated cause. As you note, the manuscript already states this causal scope.

The coverage contribution is a reproducible comparison across 153 source assets and 22 asset–operation groups that reveals differences a single-asset evaluation can miss. This contribution is distinct from the added measurement information in A1. Both contribute to the benchmark's utility.

Finally, we sincerely thank the reviewer’s feedback. We respectfully hope that the reviewer could reevaluate our work given the responses addresing your main concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.