- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces VeriEdit-Bench, a benchmark for fine-grained image editing on structured visual assets. It contains 1,740 editing cases derived from 153 source assets spanning SVG graphics, charts, web interfaces, and presentation slides. Controlled source-code modifications yield reference targets and edit masks, enabling deterministic evaluation along four axes: fidelity, preservation, localization, and magnitude. The paper evaluates eleven image editors and analyzes operation-specific performance, cross-asset differences, and sensitivity to scoring choices.
Critical strengths
-
Reproducible benchmark construction. Source-controlled edits provide explicit targets and edit regions, enabling repeatable evaluation of precisely specified changes without relying on human or VLM judgments for the core scoring.
-
Coverage of multiple structured asset types. Evaluating shared and asset-specific operations across SVG graphics, charts, web interfaces, and slides provides useful insights into performance differences that a single-domain benchmark may overlook.
-
Extensive supporting analyses. The paper includes controlled perturbations, scoring ablations, parameter-sensitivity analyses, and source-level bootstrap intervals. Reporting individual axes alongside the composite also helps expose different failure patterns.
Critical weaknesses
-
Pixel-area-based scoring may underweight semantically important errors in small regions. The benchmark targets structured assets such as charts, web interfaces, and slides, where a small change to a number, unit, minus sign, or label can substantially alter meaning. However, Preservation averages pixel agreement across the entire protected region, while Localization filters out small connected changes outside the edit region as rendering noise. Consequently, damage to small but important content may receive only a minor penalty or be omitted from Localization altogether. An output could therefore execute the requested edit while corrupting critical protected content and still receive high scores. The existing controls, which primarily examine color-group modifications and background damage, do not adequately rule out this risk. Controlled tests involving small text, numerical values, and legends, together with object-level or text-level preservation measures, would strengthen the validation.
-
The composite score exhibits a substantial floor effect, limiting its ability to distinguish partial success and incremental improvements. Appendix Table 11 reports that 52.5%–84.9% of outputs from open-weight models receive a zero composite score. Table 12 further shows that 87% of all zero-composite outputs have nonzero Localization and Magnitude but receive zero Fidelity, which forces the harmonic composite to zero. Fidelity measures agreement at a finite set of color tolerances and subtracts the unchanged-input baseline. An output can therefore move closer to the target without receiving additional credit if that improvement does not increase thresholded agreement; improvements along the other axes likewise remain invisible to the composite when Fidelity is zero. Although the individual axes retain some diagnostic information, the primary ranking metric collapses many potentially different degrees of failure into the same score, weakening the claimed fine-grained feedback. The paper would benefit from examining meaningful progress within zero-score outputs and comparing against continuous target-distance measures, beyond establishing aggregate ranking stability under threshold changes.
Questions for authors
See Weaknesses.
Q1: Pixel-area-based scoring may underweight semantically important errors in small regions.
Weakness 1
A1:
This comment identifies a valuable complementary dimension that can be reported separately: whether key text and numerical content retain their meaning. It is related to precise visual execution but asks a different question. It does not require redefining the four axes as measures of semantic correctness. Our axes use explicit visual references to measure target appearance, pixel preservation, edit extent, and change magnitude and direction (Section 3.3; Appendix B.2).
The distinction has a practical basis. A minus sign occupies few pixels but can change the meaning of a value. Assessing that importance requires recognizing the text and its context. P measures how much of the protected region retains pixel agreement. L measures spatial coverage and confinement. Neither weights pixels by the meaning of their text. Visual preservation and semantic correctness can therefore be reported separately: one measures how much of the image is preserved; the other checks whether its information remains correct.
As a supplementary control for this response, we tested your concern by constructing 57 errors in numbers, signs, units, or labels across 17 chart source assets. We first completed the requested edit, then changed one text object that should have remained intact. All constructed outputs still obtained H≥95 and P≥99; 55 had L=100. High visual agreement can therefore coexist with a local semantic error, so semantic correctness requires an additional check. These results define the interpretation of high axis scores. Such scores do not guarantee intact text meaning, but this does not invalidate their measurement of precise attributes and visual changes.
We therefore view the suggestion as a semantic-preservation dimension alongside the four axes, rather than a reason to adjust them to replace semantic judgment. Object- or text-level content checks could supply additional evidence for applications that depend on correct numbers, units, and labels. We thank you for identifying this useful direction. It extends the range of evaluated capabilities while retaining the clear meaning of the visual measurements.
Q2: The composite score exhibits a substantial floor effect, limiting its ability to distinguish partial success and incremental improvements.
Weakness 2
A2:
Using H alone compresses differences in partial success. This is why the paper reports the four-axis profile alongside H. H measures joint performance, while the axes retain the attained level of each capability. If one axis is zero, H is zero, but the other axes can still show coverage, preservation, or change-control ability. Section 3.3, Evaluation Protocol, defines this reporting scheme.
Section 4.4, Figure 5 demonstrates the distinction. Complete images and per-tolerance accuracy curves are in Appendix A.11, Figures 19–20. On the same web recoloring task, FLUX.2-dev changes the requested green elements, but its purple is shifted toward magenta. Step1X-Edit leaves the target elements green and changes unrelated page regions. Neither earns fidelity credit, and both have H=0. Their axis scores nevertheless distinguish the execution patterns:
The former shows better target coverage and color-change control but needs more accurate final colors. The latter first needs to execute the edit in the intended region. Equal H therefore does not make these capability states indistinguishable in our full reporting scheme. The axes provide interpretable feedback beyond the composite.
A supplementary continuous-distance check of the existing outputs for this response also found that some zero-score outputs are closer to the target. This further supports interpreting zero composites together with axis-level diagnostics, rather than as an absence of all progress.
Our reporting separates joint performance from partial success: H summarizes the former, while the axes expose capability differences and improvement directions. The real-output example demonstrates the value of this profile beyond reporting a composite alone.
Finally, we sincerely thank the reviewer’s feedback. We respectfully hope that the reviewer could reevaluate our work given the responses addresing your main concerns.